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1.0  SUMMARY 


Task  Order  0037,  under  the  Warfighter  Readiness  Science  &  Technology  Program  (WRSTP) 
contract,  identified  candidate  approaches  to  solve  Distributed  Mission  Operations  (DMO) 
Modeling  and  Simulation  (M&S)  deficiencies  in  the  optimal  presentation  of  infonnation  to 
decision  makers  in  multi-modal  immersive  environments.  The  scope  of  the  task  included 
developing  and  validating  system  dynamics  decision  models  and  conducting  human  factors 
experimentation  on  optimization  of  immersive  environments  for  a  wide  range  of  training 
applications.  The  tasking  also  included  identification  of  methods  for  quickly  and  accurately 
modeling,  rendering  and  displaying  physics-based  dynamic  three-dimensional  (3D)  weather, 
weapons  effects,  other  special  effects,  and  associated  visual  and  sensor  impacts  in  a  standard 
open  fonnat  for  sharing  between  networked  DMO  training  and  rehearsal  scenarios  and  locations. 
These  efforts  developed  approaches  to  train,  and  prepare  warfighters  in  immersive  environments 
with  dynamically  shared  common  weather,  weapons,  visual  and  sensor  effects  (an  identified 
capability  gap),  thereby  improving  warfighter  anticipation,  competencies,  and  associated  mission 
perfonnance.  The  WRSTP  team  has  presented  the  research  results  at  a  number  of  conferences 
and  reported  them  in  numerous  journals. 

During  this  tasking  the  WRSTP  team  also  developed  physics-based  representations  of  directed 
energy  weapons  and  their  effects  on  sensors  and  vision  and  generated  methods  of  integrating 
these  effects  into  enhanced  visual  displays.  The  team  also  demonstrated  6-degrees  of  freedom 
(6DoF)  aero  models  in  a  real-time  or  near  real-time  training  network,  and  developed  rule  sets  for 
functional  handoff  of  aero  models  from  one  level  of  complexity  to  another  to  maximize 
computational  capability.  The  above  tasking  benefits  the  United  States  Air  Force  (USAF) 
because  it  enables  warfighters  to  conduct  extremely  realistic  training  at  real  time  or  near  real 
time  against  complex  threats.  Physics-based  representations  provide  accurate  Computer 
Generated  Force  (CGF)  maneuvering  for  air  combat  training  in  networked  simulation, 
maximizing  the  utility  of  next-generation  simulation  displays,  eliminating  random  engagements 
in  training,  and  upgrade  quickly  if  new  threats  emerge. 

As  the  WRSTP  team  continued  to  execute  this  task  order,  we  identified  several  additional 
research  areas,  which  expanded  the  scope  of  work.  These  included  Entity  Modeling  and  Full 
Spectrum  Threat  Simulation  (EMTS),  Audio  Integration  Demonstration,  and  High  Resolution 
Deployable  Projector  Components  for  Simulation. 

This  final  report  summarizes  work  completed  under  Task  Order  0037  on  the  WRSTP,  contract 
number  FA8650-05-D-6502  for  the  Warfighter  Readiness  Research  Division  of  the  Air  Force 
Research  Laboratory  (AFRL)  (711  HPW/RHA). 

2.0  RESEARCH  AREAS 

This  section  details  the  following  research  areas:  Robust  Immersive  Decision  Environments 
Research  (RIDER),  Rehearsal  Enabling  Simulation  Technologies  (REST),  Entity  Modeling  and 
Full  Spectrum  Threat  Simulation,  Audio  Integration/Demonstrations,  and  Ultra-High  Resolution 
Deployable  Projector  Components  for  Simulation. 
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2.1  Robust  Immersive  Decision  Environments  Research 

The  RIDER  research  effort  identified  methods  and  fidelity  requirements  for  training  in 
immersive  decision-making  environments.  The  WRSTP  team  investigated  three  different  types 
of  decision-making  models:  decision  priming,  Recognition  Primed  Decisions  Model  (RPD),  and 
implicit  learning  for  intuitive  decision  making.  Decision  priming  refers  to  a  situation  in  which 
multiple  sources  of  information  that  are  presented  to  an  observer  are  distributed  in  time,  and  the 
observer  is  asked  to  make  a  speeded  decision  as  to  whether  to  execute  one  type  of  response  or 
another.  Decision  priming  occurs  when  initial  information  suggests  the  expectation  that  a  given 
decision  is  appropriate,  and  that  expectation  speeds  up  decision-making.  The  RPD  model 
(Patterson,  Pierce,  Bell,  Andrews,  and  Winterbottom  ,  2009;  Patterson,  Fournier,  Pierce, 
Winterbottom,  and  Tripp,  2009;  Patterson,  Fournier,  Pierce,  Winterbottom,  and  Tripp,  2009) 
states  that  decision  making  in  naturalistic  contexts  entails  a  situational  pattern-recognition 
process,  which  if  subsequent  expectancies  are  confirmed,  lead  the  decision  maker  to  render  a 
decision  to  engage  in  a  given  course  of  action. 

The  implicit  learning  effort  began  by  exploring  whether  findings  from  the  basic  literature  (Reber, 
1967;  Reber,  1989)  for  implicit  statistical  learning  of  static  information  can  be  replicated  by 
employing  simulated  tactical  scenarios  in  an  immersive  environment  where  information  will  be 
dynamic,  more  realistic  and  therefore  more  complex.  Implicit  statistical  learning  refers  to  the 
learning  of  statistical  regularities  in  a  pattern  of  stimulation  without  a  deliberate  conscious  effort 
(Patterson,  Pierce,  Bell,  and  Klein,  2009;  Patterson,  2010).  It  has  been  suggested  that  this  is  a 
means  by  which  the  foundation  for  the  intuitive  pattern-recognition  is  developed.  This  intuitive 
process  greatly  aids  in  robust  decision  making  in  dynamic  environments  where  information  is 
limited  and  time  constraints  exist.  The  possibility  that  implicit  learning  can  lead  to  successful 
intuitive  decision  making  made  it  likely  that  the  development  of  such  decision  making  could  be 
studied  by  inducing  implicit  learning  with  high-fidelity  environmental  cues  and  patterns. 

Through  a  series  of  experiments,  various  training  methods  and  requirements  were  explored. 

These  include  how  best  to  design  and  present  infonnation  for  implicit  learning  within  a  training 
environment  (Patterson,  Pierce,  Boydstun,  Park,  Shannan,  and  Tripp,  2010;  Patterson,  Pierce, 
Boydstun,  Park,  Shannan,  and  Tripp,  2010)  determining  how  diversity  and  organization  of  the 
training  stimuli  affect  learning  and  whether  this  infonnation  can  be  transferred  to  another  domain 
(Winterbottom,  Patterson,  Fournier,  Pierce,  Williams,  Amann,  2009),  how  algorithmic 
complexity  (length  and  complexity  of  the  algorithm)  affects  the  acquisition  of  infonnation  and 
how  implicitly  learned  infonnation  is  retained  over  time  (Boydstun,  Patterson,  Pierce,  Park,  and 
Shannan,  2010;  Boydstun,  Patterson,  Pierce,  Park,  and  Tripp,  2011).  A  paradigm  was  developed 
to  explore,  and  ultimately  did  demonstrate,  the  unconscious  and  non-volitional  acquisition  of 
information  relating  to  the  patterns  of  image  sequences  in  a  flight-simulated  environment.  The 
enhanced  understanding  of  robust  decision-making  processes,  which  resulted  from  this  work, 
will  help  identify  user  cuing  needs  and  technology  requirements  to  optimally  train  decision 
makers  in  multi-modal  immersive  environments.  The  effort  will  provide  to  current  and  future 
warfighters  the  capability  to  experience,  train,  and  rehearse  in  immersive  environments  with  cues 
necessary  for  robust  decision  making. 

2.1.1  Methods,  Assumptions,  and  Procedures 

For  the  intuitive  decision  making  experiments,  we  investigated  whether  naturalistic,  intuitive 
(pattern-recognition-based)  decision  making  can  be  developed  via  implicit  statistical  learning  in 
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a  simulated  real-world  environment.  The  environment  was  composed  of  object  sequences  (e.g., 
vehicle,  house)  placed  on  terrain  imagery  presented  in  perspective  view,  and  the  order  of  the 
objects  in  the  sequences  was  defined  by  different  paths  taken  through  a  finite-state  algorithm. 
Images  and  the  terrain  were  generated  using  MetaVR  Virtual  Reality  Scene  Generator  (VRSG) 
and  were  presented  on  a  50”  display  with  a  resolution  of  1920  x  1080.  The  environment  was 
presented  dynamically  which  induced  a  sense  of  simulated  locomotion  through  the  scene  as  the 
participants  viewed  the  object  sequences.  The  sequences  were  either  generated  randomly  or  via  a 
statistical  algorithm.  During  training,  subjects  passively  viewed  a  sample  of  target  sequences 
from  the  total  number  of  sequences  produced  by  the  algorithm.  During  test,  subjects  viewed 
another  sample  of  the  remaining  target  sequences  randomized  with  an  equal  number  of  randomly 
generated  sequences,  and  were  asked  to  make  intuitive  decisions  about  whether  the  sequence 
presented  represented  one  from  the  algorithm  or  was  random. 

2.1.2  Results  and  Discussion 

The  principal  results  of  the  decision  making  research  effort  shows  that  passive  exposure  to 
spatio-temporal  patterns  in  a  simulated  real-world  environment  can  induce  a  level  of  implicit 
learning  that  is  more  than  sufficient  for  creating  a  foundation  for  intuitive  (pattern-recognition- 
based)  decision  making.  This  suggests  implicit  learning,  along  with  working  memory,  does  play 
a  role  in  the  development  of  intuitive  decision  making  and  implicit  learning  can  be  a  vehicle  for 
developing  intuitive  decision  making  in  a  simulated  real-world  environment. 

In  our  early  experiments,  we  investigated  the  acquisition  of  intuitive  decision  making  via  the 
process  of  implicit  learning.  Participants  passively  viewed  structured  sequences  of  objects  or 
letter  strings  presented  either  on  a  dynamic  terrain  seen  in  perspective  view  or  on  a  static  flat 
display.  We  also  had  participants  memorize  structured  strings  of  letters  presented  on  a  static  flat 
display.  Following  training,  participants  were  tested  for  implicit  learning  by  making  intuitive 
judgments  of  novel  structured  sequences  versus  random  sequences.  The  results  of  this  effort 
showed  that  implicit  learning  occurred  easily  across  all  conditions  and  there  was  no  significant 
perfonnance  advantage  to  one  viewing  condition  over  another.  This  study  has  implications  for 
the  training  of  intuitive  decision  making  insofar  as  it  demonstrates  that  cues  can  be  passively  and 
effectively  acquired  from  exposure  to  different  artificial  displays. 

As  a  follow-up,  we  investigated  the  effects  of  articulatory  suppression,  training  block  size,  and 
complexity  within  the  immersive  implicit-learning  paradigm.  These  results  showed  articulatory 
suppression  and  pattern  complexity  impaired  decision  making,  but  block  size  did  not.  Perfonning 
an  articulatory  suppression  task  degraded  decision  making  because  the  task  occupied  explicit 
cognitive  resources,  which  are  likely  linked  to  working  memory.  Training-sequence  length 
interacted  with  algorithmic  complexity  such  that  performance  was  best  when  training-sequence 
length  was  long  and  the  algorithm  was  simple,  and  when  training-sequence  length  was  short  and 
the  algorithm  was  complex.  Thus,  it  was  determined  that  when  training  intuitive  decision 
making,  training-sequence  length  should  be  matched  to  algorithmic  complexity. 

Implicit  learning,  which  entails  the  largely  unconscious  learning  of  dynamic  statistical  patterns 
and  features,  is  a  ubiquitous,  robust  phenomenon  that  likely  occurs  in  most,  if  not  all,  tasks  in 
which  individuals  engage  throughout  their  lives.  Thus,  implicit  learning  is  likely  to  occur,  in 
parallel  fashion,  when  individuals  develop  expertise  in  a  given  content  domain.  This  invites  the 
speculation  that  the  enhancement  of  implicit  learning  may  assist  in  the  acquisition,  retention,  and 
transfer  of  expertise. 
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2.2  Rehearsal  Enabling  Simulation  Technologies 

REST  is  an  AFRL  program  actively  engaged  to  improve  DMO  simulator  database  development 
processes.  REST  established  a  prototype  DMO  database  generation  and  modification  system  to 
aid  evaluation  of,  and  identify  improvements  to,  the  end-to-end  DMO  database  generation 
process  supporting  ah  DMO  players.  This  effort  comprised  the  development  of  the  database 
generation  system  as  well  as  the  demonstration  of  databases  that  were  generated  using  the  REST 
system. 

The  REST  program  investigates  novel  source  data  and  software  tools  to  improve  the  DMO 
database  generation  process.  It  also  participates  in  the  development  of  database  standards  to 
improve  database  reusability,  fidelity,  and  correlation.  The  REST  database  generation  system 
consists  of  software  tools  and  processes  to  be  transitioned  to  ACC  for  the  rapid  development  of 
high  fidelity  visual  and  sensor  simulator  databases  for  use  in  DMO  training  and  rehearsal.  These 
databases  will  be  stored  in  standard  industry  data  fonnats  which  can  be  easily  shared  and  reused 
by  all  DMO  participants. 

2.2.1  Methods,  Assumptions,  and  Procedures 

Using  the  general  REST  DMO  database  generation  process,  the  REST  team  was  able  to  build 
databases  and  orchestrate  the  building  of  other  databases  in  different  runtime  image  generator 
(IG)  fonnats  with  a  high  degree  of  correlation  by  building  each  using  the  same  source  dataset 
package.  A  variety  of  commercial  data  manipulation  software  tools  which  were  able  to  ingest  the 
same  data  formats  (Digital  Terrain  Elevation  Data  [DTED],  Geospatial  Tagged  Imagery  File 
Format  [GeoTIFF],  Environmental  Systems  Research  Institute  [ESRI]  Shape,  and  Presagis 
OpenFlight)  were  used. 

REST’s  source  data  is  stored  in  Air  Force  Common  Dataset  (AFCD)  fonnats  enabling  ingestion 
by  most  or  all  database  generation  toolsets.  Elevation  data  is  stored  in  DTED  fonnat,  and  ground 
imagery  is  in  GeoTIFF.  Presagis  TerraVista,  TerraSim’s  TerraTools,  ESRI  ArcGIS,  and 
MetaVR’s  Metadesic  Compiler  can  read  these  formats  to  generate  tenain  databases.  For  cultural 
features,  we  used  models  in  Presagis’  OpenFlight,  which  is  easily  converted  into  most  other 
runtime  formats.  To  describe  the  placement  of  these  models  we  used  ESRI  Shape  files.  A  shape 
file  can  be  viewed  and  edited  in  ArcGIS,  and  can  be  used  to  place  models  in  databases  created  by 
TerraVista  or  TerraTools.  MetaVR  TerrainTools  can  convert  shape  files  into  its  culture  file 
format,  which  can  be  used  to  place  models  in  a  MetaVR  database.  For  some  of  the  REST 
demonstrations,  we  used  DTED  Level  2  and  5  meter  and  60  centimeter  imagery  from  REST’s 
Western  Ranges  Dataset  (WRDS),  lm  imagery  provided  by  the  USAF’s  B-l  training  system 
program,  building  models  provided  by  the  UK  Ministry  of  Defense  (MoD),  and  buildings  and 
vehicle  models  from  various  other  sources,  most  notably  the  Army’s  Synthetic  Environment 
(SE)  Core  program. 

2.2.2  Results  and  Discussion 

In  order  to  demonstrate  the  capabilities  of  the  REST  system  developed  under  this  effort,  multiple 
databases  were  developed  under  a  requirement  for  the  Interservice/Industry  Training,  Simulation 
and  Education  Conference  (I/ITSEC)  2010.  The  requirement  involved  building  a  scenario  in 
which  friendly  forces  would  deploy  from  a  forward  operating  base  (FOB)  and  escort  a  United 
Nations  (UN)  convoy  to  a  village,  where  they  would  be  ambushed  by  ground  forces  and  a 
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helicopter.  The  REST  team  built  correlated  terrain  databases  for  each  of  the  five  simulators 
involved  that  collectively  resembled  the  Middle  East  and  included  an  FOB  and  a  village. 

The  team  encountered  unanticipated  problems  during  the  course  of  building  the  databases.  Many 
of  these  resulted  from  a  lack  of  familiarity  with  the  tools  involved,  or  minor  issues  within  the 
tools  themselves.  There  also  were  correlation  issues  in  the  OpenFlight  models  resulting  from 
using  them  in  runtime  systems  not  designed  for  this  use,  or  from  converting  them  into  different 
formats.  Most  of  these  problems  were  solved  and  the  solutions  were  documented. 

Although  OpenFlight  is  a  standard  fonnat  for  storing  3D  models,  OpenFlight  also  has  numerous 
methods  of  handling  things  like  lighting  and  infrared  (IR)  effects.  The  following  is  a  list  of  some 
of  the  attribute  problems  encountered  with  reusing  OpenFlight  models  from  other  programs:  IR 
polygons,  nighttime  texturing,  level  of  detail,  primary  color  of  textures,  buildings  with  footprints, 
and  texture  drawing  problems. 

We  chose  to  build  the  MetaVR  database  first,  because  of  how  easy  it  is  to  place  models  in 
MetaVR  and  then  convert  the  resulting  culture  file  to  a  shape  file  that  can  be  used  to  build  the 
other  databases.  However,  it  would  also  be  possible  to  create  a  MetaVR  database  using  a  shape 
file  from  another  database.  TerrainTools  can  convert  a  shape  file  to  a  culture  file  just  as  easily. 
Given  an  OpenFlight  database,  for  example,  along  with  the  source  data  used  to  generate  it  and  a 
shape  file  describing  the  placement  of  its  cultural  features;  one  could  easily  generate  a  correlated 
MetaVR  database  from  that. 

In  addition,  many  other  software  tools  exist  which  can  produce  similar  results.  Presagis 
TerraVista  can  also  produce  OpenFlight  and  VBS2  terrain  databases.  Global  Mapper  can  be  used 
to  edit  and  reproject  imagery.  MetaVR  databases  can  be  built  without  the  use  of  ArcGIS  and 
TerrainTools,  but  they  help  a  great  deal  in  visualizing  and  selecting  the  data  to  be  used  in 
generating  a  database.  TerrainTools’  ability  to  convert  shape  files  to  and  from  culture  files  also 
makes  generating  correlated  databases  a  great  deal  easier. 

The  demonstration  databases  built  and  provided  for  I/ITSEC  2010  represents  only  one  success  in 
the  development  of  the  REST  Database  generation  process  and  generation  of  multiple  databases 
using  this  process.  The  demonstration  was  a  success  and  also  produced  some  lessons  learned  in 
the  creation  of  the  databases.  The  lessons  learned,  as  described  above,  will  be  integrated  into 
future  database  development  processes  and  used  to  further  refine  and  standardize  this  process. 

2.3  Entity  Modeling  and  Full  Spectrum  Threat  Simulation 

Entity  Modeling  is  developing  modeling  and  simulation,  physics,  and  computational  methods  to 
provide  optimal  real-time  computer-generated  forces  interactions,  and  models  of  advanced  and 
emerging  threats,  sensors,  emitters,  communications,  and  data-links.  The  WRSTP  team 
developed  and  modified  software  tools  such  as  eXpert  Common  Immersive  Theater 
Environments  (XCITE)  to  implement  the  use  of  user-defined  triggers  for  executing  tactics  and 
scripts  and  operating  sensors  and  radios. 

2.3.1  Methods,  Assumptions,  and  Procedures 

Modeling  highly  complex  entity  interactions  with  high  precision  is  challenging  in  distributed 
training  simulations.  During  this  effort,  the  team  developed  physics-based  directed  energy  and 
other  threat  models,  such  as  missiles,  and  integrated  them  into  the  DMO  test  bed.  We  maintained 
compatibility  with  existing  simulation  facilities  and  assessed,  validated,  and,  documented  the 
resulting  training  capabilities  of  the  combined  systems  when  applicable. 
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2.3.2  Results  and  Discussion 


Research  and  development  work  on  this  effort  produced  numerous  results  as  outlined  in  the 
sections  below. 

2.3.2. 1  Missile  Server 

The  WRSTP  team  integrated  advanced  AIM- 120,  6DoF  real-time  or  near  real-time  weapon 
models  developed  under  a  Cooperative  Research  and  Development  Agreement  (CRADA)  with 
Raytheon  into  DMO  and  live-virtual-constructive  (LVC)  environments  for  evaluation  of  training 
utility. 

2.3.2.2  Directed  Energy 

As  a  standard  that  is  currently  in  development,  the  DMO  environment  previously  did  not  support 
Directed  Energy  (DE)  weapons  under  the  DIS  standard.  After  receiving  a  draft  of  the  new 
standard  from  developers  at  AFRL’s  Directed  Energy  Directorate,  we  recognized  that 
incorporating  DE  weapons  into  XCITE  was  a  two-part  development  project.  We  had  to  add  the 
ability  to  send  and  process  properly  formatted  DE  protocol  data  units  (PDUs)  to  the  XCITE 
DISNET  architecture.  In  addition,  these  weapons  had  to  be  given  special  processing  within  the 
XCITE  core  code,  as  they  do  not  have  the  same  flight  and  destruction  physics  as  traditional 
kinetic  weapons.  We  solved  the  first  part  by  investigating  the  current  method  for  sending  and 
processing  other  types  of  messages  and  adapting  them  for  these  new  DE-specific  fields.  The 
second  part  required  laying  out  a  way  to  fit  these  massless,  speed  of  light  weapons  into  the 
current  weapon  structures.  We  added  new  fields  and  exceptions  in  the  code  to  handle  them  as 
well  as  setting  aside  a  procedure  for  processing  damage  inflicted  by  DE  weapons  based  on  a 
simple  dwell  time  model.  After  testing  in  multiple  environments  onsite,  a  DE-enabled  version  of 
XCITE  was  delivered  to  customers  at  the  AFRL  DE  Simulation  Lab  at  Kirtland  AFB  prior  to  the 
ACE  10  (Advanced  Concepts  Event  2010)  event  in  April  2010.  Viewed  by  upper  members  of  the 
Air  Force  and  Pentagon,  the  Chief  of  Staff  of  the  Air  Force  recognized  XCITE  for  its 
involvement  in  the  event. 

The  WRSTP  team  completed  the  design,  coding,  and  integration  effort  that  provided  an 
automated  handoff  of  the  responsibility  for  computing  entity  damage  status  during  a  DE 
engagement.  Previously,  XCITE  always  computed  and  published  damage  status.  The  handoff 
gives  the  computation  task  to  the  L-3  Orlando  Physics-Based  Environment  Generator  (PBEG) 
application.  Now  PBEG  may  provide  XCITE  with  the  information  it  needs  to  publish  damage 
status  to  all  applications  in  the  DMO  network. 

2.3.2.3  Entity  Count  and  Processing  Capacity 

WRSTP  engineers  investigated  methods  to  increase  entity  count.  A  method  was  tried  which 
employed  hyper-threading  to  increase  the  number  from  185  to  205  XCITE  player  entities  per  60 
Hz  frame.  Continued  work  would  prove  useful  because  a  multicore,  multiprocessor  computer 
could  process  high-fidelity  player  models  at  60  Hz  for  up  to  at  least  1,000  entities  without 
experiencing  frame  overruns.  The  key  is  to  redesign  the  XCITE  software  to  make  better  use  of 
threading  and  object  oriented  design.  The  current  design  is  process  oriented  and  the  computers 
that  XCITE  typically  runs  on  at  the  research  site  are  underutilized  when  there  are  multiple 
processors  available,  which  the  XCITE  routines  do  not  use. 

The  WRSTP  team  completed  a  majority  of  the  development  of  application  software,  which  now 
provides  the  ability  for  AFRL’s  XCITE  and  Threat  Instructor  Operator  Station  (IOS)  tools  to 
function  simultaneously  on  the  same  computer.  Previously,  the  engineers,  research  scientists,  and 
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instructor  pilot  community  were  required  to  obtain  and  use  at  least  two  computers  to  accomplish 
any  training  event  that  used  XCITE. 

2.3.2.4  Database  Commonality 

An  earlier  growth  path  of  the  Next  Generation  Threat  System  (NGTS)/XCITE  research  and 
development  had  used  differing  entity  model  naming  conventions  for  each  of  the  various  security 
classification  levels  of  the  CGF  application’s  databases.  This  difference  in  the  names  of  models 
proved  problematic  when  creating,  reusing,  merging,  or  sharing  model  data.  Where  possible,  the 
structure  and  form  of  data  tables  and  parameters  should  remain  common  despite  differences  in 
security  classification  level  of  the  data.  The  WRSTP  team  have  successfully  modified  the 
software  functions  and  database  for  more  commonality  at  ah  the  security  levels.  Developers 
found  that  this  effort  cleared  up  several  issues  and  errors  that  were  inherent  in  the  previous 
design. 

2.3.2.5  Scenario  Overlay 

The  training  research  community  expressed  the  need  for  the  XCITE  CGF  to  load  additional 
previously  saved  scenario  data.  After  the  user  loaded  an  initial  scenario  and  began  the  training 
event,  they  could  not  add  further  saved  scenario  information  without  clearing  the  currently 
running  scenario.  We  developed  the  ability  for  an  instructor  operator  to  easily  generate  several 
new  entities  by  selecting  a  saved  file.  Now  the  training  environment  can  be  quickly  expanded  to 
meet  immediate  needs  without  being  -trapped”  in  the  bounds  of  capability  of  just  a  single 
scenario  data  file.  Future  growth  plans  should  include  a  way  to  automatically  script  the  scenario 
overlay  function  to  reduce  operator  workload  during  the  creation  of  scenarios,  this  ability  could 
reduce  effort  needed  when  the  operator  is  reusing  portions  of  scenarios. 

2.3.2.6  Jammers 

The  WRSTP  team  developed  XCITE  sensor/emitter  models  to  represent  advanced  physic-based 
radar  jamming  systems.  The  CGF  uses  jammer  emitter  identification  data  tables  to  apply  specific 
techniques  to  a  target’s  radar.  New  algorithms  developed  model  blinking  noise,  cooperative 
Doppler  noise,  and  others.  The  CGF  applies  the  jamming  to  networked  training  devices  by 
publishing  standard  DIS  packets. 

The  enhanced  Threat  IOS  parses  ah  the  information  received  in  a  DIS  Emission  PDU.  Received 
parameters  display  status  to  the  research  scientists  and  instructor/operators.  This  includes  ah 
track  files  and  jam  tracks.  In  a  DMO  network  environment,  the  feedback  provided  shows  which 
emitters  are  jamming  which  sensors  and  weapons,  with  which  jamming  class  and  technique.  The 
enhancement  benefits  the  provider  of  the  training  scenario  because  users  previously  did  not 
observe  accurate  status  for  ah  sensor  systems  and  sensor  beams,  and  were  not  able  to  see  or 
control  jammer  effectiveness  and  jammer  emitter  IDs. 

The  XCITE  application  to  simulate  towed  decoys  was  primarily  developed  on  a  different  task 
order.  However,  this  TO  0037  effort  included  work  to  update  the  Threat  IOS  tool  to  allow  the 
end-users  to  deploy  and  sever  the  decoys,  visualize  the  devices,  and  control  the  jamming 
technique  of  the  decoys. 

2.3. 2.7  Weapon  Employment,  Doctrine,  and  Tactics 

The  WRSTP  team  enhanced  the  XCITE  CGF  to  accept  and  use  weapon  employment,  doctrine, 
and  tactics  (EDT)  parameters  provided  by  instructors/operators  after  a  training  scenario 
commenced.  Previously,  users  defined  static  values  in  a  database  and  these  remained  constant 
throughout  the  mission  or  event.  The  operational  training  community  desired  to  adjust  EDT 
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parameters  -on  the  fly”  by  easily  and  quickly  entering  or  picking  new  values  while  the  mission 
was  running.  Now  that  the  CGF  has  been  upgraded,  the  remaining  development  involves  making 
the  corresponding  enhancement  to  the  Threat  IOS  application.  The  XCITE  graphical  user 
interface  (GUI)  should  be  modified  to  show  the  user’s  current  EDT  status  for  a  selected  player 
entity/target  entity  pairing.  The  Threat  IOS  should  give  the  option  to  change  EDT  parameters  at 
anytime  or  to  revert  to  the  default  values  using  a  simple  user  control. 

2.3.2.8  Triggers 

The  WRSTP  team  introduced  the  capability  in  the  XCITE  CGF  to  employ  user-defined  triggers 
that  cause  simulated  entities  to  act  and  behave  in  a  more  versatile  manner  that  may  often  appear 
more  lifelike  to  the  student/trainee.  Research  should  continue  to  put  triggers  under  the  control  of 
a  user-definable  script.  This  will  vastly  reduce  operator  workload  and  produce  a  realistic  training 
environment. 

2.3.2.9  Transfer  of  Control 

WRSTP  engineers  explored  methods  to  enable  XCITE  to  automatically  transfer  control  of  player 
entities  between  the  various  CGFs  in  a  DMO  network. 

2.3.2.10  NAVAIDS 

WRSTP  engineers  developed  XCITE  to  simulate  navigational  aid  systems  (NAVAIDS).  Users 
can  easily  create,  move,  and  delete  entities.  Frequency  and  transmitter  state  are  user  editable.  The 
tool  includes  a  database  of  known  Tactical  Air  Navigation  Systems  (TACANs).  Transmitter 
PDUs  and  entity  state  PDUs  are  published  for  all  NAVAIDS  that  are  created  by  or  controlled 
internal  to  XCITE.  Additional  software  development  work  is  necessary  to  achieve  conformance 
to  DIS  standards.  The  extra  effort  should  include  modifications  to  the  XCITE  network  interface 
unit  (NIU)  to  further  develop  support  of  the  standard  identification  friend  or  foe  (IFF)/air  traffic 
control  (ATC)/NAVAIDS  PDU.  This  enhancement  was  not  completed  during  the  course  of  this 
task  order  because  of  limited  funding  for  software  engineering  and  development. 

2.3.2. 1 1  Manual  Control 

Scenario  providers  in  DMO  events  want  more  manual  control  of  constructive  entities  than  is  in 
the  current  version  of  AFRL’s  XCITE  application.  Research  and  development  should  continue  to 
explore  ways  to  give  users  the  ability  to  simply  select  a  player  entity  and  quickly  choose  which 
systems  they  want  to  control  while  allowing  the  CGF  program  to  continue  simulating  the 
algorithms,  modes,  and  effects  that  the  operator  wishes  to  remain  operating  automatically  and 
under  the  control  of  other  players  or  scripts.  WRSTP  contractors  recommend  integrating  AFRL’s 
DIS  network  enhancements  to  the  XPlane  tool  with  the  classified  weapon,  sensor,  and 
emitter/jammer  routines  provided  by  XCITE.  Selected  entities  may  then  be  published  to  the 
network  with  DIS  packets  being  generated  by  both  applications  that  split  their  responsibility  for 
modeling  the  players  which,  effectively,  will  be  constructive  entities  behaving  with  some 
characteristics  of  virtual  cockpits.  Continued  R&D  in  this  area  would  leave  the  complex, 
classified  data  and  routines  processing  within  XCITE  while  the  flight  parameters  and 
characteristics  would  be  developed  and  enhanced  within  the  XPlane  program.  Additionally,  users 
would  get  the  benefit  of  realistic  control  of  flight  with  a  joystick  instead  of  having  to  use  a  mouse 
and  keyboard,  and  will  have  a  3D  view  not  currently  available  in  the  Threat  IOS  application.  The 
outcome  of  this  proposed  effort  would  mean  XCITE  will  publish  the  necessary  infonnation, 
which  the  XPlane  graphics  routines  use  to  simulate  cockpit  display  screens  and  the  head  up 
display  (HUD).  Therefore,  entity  modeling  and  control,  as  well  as  scenario  visualization  will  be 
distributed  among  tools  that  may  be  better  suited  for  the  tasks. 
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2.3.2. 12  Identification  Friend  or  Foe 

WRSTP  engineers  enhanced  AFRL’s  XCITE  to  model  IFF  mode  4  rollover  and  also  modified 
the  Threat  IOS  application  to  present  current  IFF  status  for  a  given  player  entity  in  ways  that  are 
better  for  the  users. 

2.3.2. 13  Scripted  Tactics  and  Behavior 

The  WRSTP  team  explored  usability  of  GUIs  for  building  and  running  training  scenarios,  which 
have  constructive  entity  tactics  and  behavior  executed  under  control  of  a  script. 

The  primary  goal  of  this  research  was  to  develop  a  more  efficient  way  to  build  scripts. 
Previously,  the  government’s  XCITE  application  provided  the  user  only  one  way  to  build  an 
entity  script.  An  Access  database  accepted  user  inputs  and  stored  the  script  information  in  a 
scenario  initialization  data  file.  The  data  was  read  into  the  application’s  memory  space  when  the 
user  selected  a  predefined  scenario.  We  defined  entity  characteristics  and  initial  position,  along 
with  script  parameters,  before  the  training  event  began  and  they  could  not  be  changed  while  the 
loaded  scenario  was  still  executing. 

End-users  in  the  operational  training  community,  as  well  as  research  scientists  and  the  primary 
investigators  for  this  task,  found  the  existing  scripting  tools  and  techniques  to  have  little  value. 
This  was  true  because  the  capabilities  of  this  application  to  run  scripts,  and  the  existence  of  a 
user  front-end  on  the  database  tool,  was  not  adequate  because  they  required  experienced,  highly 
trained  individuals  with  computer  science,  physics,  and  engineering  backgrounds  to  perform  a 
very  complex  and  tedious  task  which  most  people  would  not  or  simply  could  not  do  themselves. 

A  secondary  objective  was  to  reduce  or  eliminate  the  need  to  enter  and  store  duplicate  script 
parameters  for  pairs  or  groups  of  constructive  entities  assigned  to  follow  the  exact  same  script. 
Users  desire  the  ability  to  have  a  vast  library  of  predefined,  customized  scripts,  which  they  may 
select  at  any  time  and  assign  an  entity  or  a  group  to  follow  or  execute.  Additionally,  users  want 
to  provide  extra  parameters  unique  to  any  given  combination  of  an  entity  and  a  script.  For 
instance,  a  script  containing  an  instruction  to  attack  a  target  need  not  define  exactly  which  target 
until  an  asset  entity  is  assigned  to  execute  the  script.  At  the  script  assignment  stage,  end-users 
want  to  be  able  to  define  target  type,  position,  and  type  of  ordnance  to  deliver.  For  this  example, 
one  single  script  can  be  used  for  four  airborne  assets.  Each  may  have  its  own  unique  target 
and/or  ordnance  selection,  but  all  other  script  parameters  may  be  considered  common. 

Another  important  part  of  this  research  was  to  develop  a  way  for  XCITE  users  to  visualize 
scripts  while  player  entities  are  executing  them.  Looking  at  an  IOS  terminal,  the  instructors  and 
scenario  providers  want  to  see  where  an  entity  is  within  its  assigned  script.  They  want  to  quickly 
view  which  instructions  have  completed,  which  one  is  currently  being  executed,  and  what  is  to 
come  and  under  what  conditions  those  will  occur.  Users  need  to  be  able  to  anticipate  what  will 
soon  transpire  or  occur  in  a  scripted  training  event,  and  they  want  to  be  able  to  do  this  without 
having  to  memorize  the  whole  event  beforehand. 

Finally,  it  is  necessary  for  one  to  be  able  to  build,  adjust,  or  otherwise  edit  a  script  at  any  time 
during  a  simulation  event  or  scenario  that  is  running.  Likewise,  anyone  should  be  able  to 
reassign  an  XCITE  player  entity  to  cease  execution  of  a  script  and  then  follow  a  different  script 
or,  as  in  the  example  above,  change  ordnance  type  in  a  script  that  has  an  instruction  to  attack. 
Users  want  to  have  an  assigned  entity  be  able  to  leave  a  script,  perform  some  other  user-defined 
task  or  operation,  and  then  easily  return  to  the  previously  assigned  script  at  the  point  where  it  left 
off  or  enter  back  in  at  a  different  instruction  in  the  script. 
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The  WRSTP  team  made  enhancements  to  the  Threat  10  S  to  improve  the  building,  editing,  and 
assigning  of  XCITE  scripts.  We  created  new  dialogs,  menus,  and  mouse  actions  to  support  this 
effort.  We  developed  new  XCITE  script  symbology  and  it  mirrors  much  of  what  Massachusetts 
Institute  of  Technology  (MIT)  created  for  the  educational  computer  program  named  Scratch.  We 
built  upon  existing  DIS  network  interfaces  between  Threat  IOS  and  XCITE  to  exchange  script 
editing  messages  and  status  infonnation.  These  network  interfaces  do  need  more  work  to  iron  out 
minor  software  bugs  as  well  as  complex  issues  with  the  design.  We  modified  the  scenario¬ 
spawning  algorithm  in  the  XCITE  application  to  use  a  common  script  library  and  entity-specific 
script  parameters.  The  engineering  team  recommends  continued  research  and  development  of  the 
necessary  software  enhancements  to  improve  usability  of  XCITE  entity  scripts. 

2.3.2. 14  Wheel  Tactic 

Using  input  provided  by  JTAC  SMEs,  the  WRSTP  team  resumed  developing  the  Wheel  tactic 
that  was  first  introduced  in  the  AFRL’s  XCITE  software  source  code  in  2005  but  not  made  ready 
for  use.  We  designed,  coded,  and  integrated  all  new  GUIs  to  allow  scenario  providers  to 
command  a  group  of  constructive  entities  to  fly  the  wheel  tactic.  Further  research  is  needed  to 
iron  out  software  design  issues  in  XCITE  so  that  the  aircraft  can  drop  its  ordnance  load  and 
return  to  the  wheel  pattern,  as  expected. 

2.3.2. 15  Lifeform  Interactions 

The  WRSTP  team  developed  within  the  XCITE  database  a  new  weapon  modeled  on  an  M-16 
rifle.  They  also  modified  the  database  to  attach  the  M-16  weapon  to  a  soldier  lifeform  entity. 
Now,  instructors  operating  a  simulated  mission  may  create  and  control  multiple  XCITE  soldier 
entities.  We  modified  the  application’s  tactics  algorithms  to  provide  the  operator  the  means  to 
manually  command  soldiers  to  attack,  shoot,  damage,  and  kill  other  opposing  lifeform  entities. 
We  enhanced  the  targeted  soldiers’  tactics  to  react  to  the  detonations  or  bullet  impact  points 
when  land-based  platforms  or  other  lifefonns  attack  soldiers  on  the  ground.  WRSTP  engineers 
developed  the  lifeform  tactics  to  cause  the  soldiers  to  move  away  from  the  fire  and  then  react  by 
returning  fire. 

WRSTP  engineers  also  modified  the  XCITE  application  to  accept  instructor  commands  to  cause 
lifefonns  to  board  a  given  vehicle  or  disembark  the  vehicle.  Once  onboard,  the  vehicle  entity  can 
be  commanded  by  the  instructor  to  move  under  control  to  another  location  while  the  lifeforms 
remain  onboard  for  the  duration  of  the  trip.  WRSTP  engineers  developed  the  application  so  that 
entities  can  be  boarded  one  at  a  time  (as  many  as  four  per  vehicle).  When  they  disembark,  all 
lifefonns  react  at  once  when  the  instructor  gives  the  command  to  get  out  of  the  vehicle. 

2.3.2.16  Patrol  Tactic 

This  tasking  included  work  performed  by  the  WRSTP  team  to  develop  the  XCITE  tool  to 
simulate  a  new  patrol  tactic  for  ground-based  lifeforms  and  vehicles.  The  scenario  provider  uses 
Threat  IOS  to  specify  the  speed  and  direction  of  the  patrol,  the  location  and  the  assets,  as  well  as 
the  number  of  laps  through  the  assigned  maneuver  area  before  the  entities  continue  with  the  next 
user-defined  action. 

2.3.2. 1 7  Rules  of  Engagement 

WRSTP  engineers  considered  options  to  enhance  the  XCITE  tool  to  improve  user-defined  player 
entity  rules  of  engagement.  However,  the  work  did  not  culminate  in  a  change  to  the  software 
baseline.  This  work  should  be  continued  because  the  user  community,  in  recent  meetings,  did 
offer  infonnation  that  illustrated  the  value  of  this  needed  feature.  A  -permission  to  engage” 


10 

DISTRIBUTION  A.  Approved  for  public  release;  distribution  unlimited.  (Approval  given  by 
88  ABW/PA,  88ABW-201 2-1 090,  5  Mar  1 2.) 


option  should  be  given  to  player  entities  created  by  or  controlled  internal  to  XCITE.  Command 
and  control  rule  sets  should  be  easy  to  setup  and  modify;  this  additional  work  should  be 
accomplished  as  time  and  funds  allow  the  team  to  continue  their  effort. 

2.3.2. 18  Route  Maneuver  Enhancements 

Responding  to  feedback  provided  by  end-users,  the  WRSTP  team  collaborated  to  develop  the 
tool’s  ability  to  fly  or  otherwise  move  constructive  entities  along  a  predefined  route.  We 
modified  the  XCITE  program  to  accept  user  inputs  and  apply  them  to  the  route  maneuver  so  that 
single  entities  or  groups  of  players  would  perform  a  variety  of  actions.  The  legacy  software 
routines  still  offer  the  ability  to  begin  a  route  at  a  selected  waypoint  and  go  along  the  remainder 
of  the  waypoints  until  arriving  at  the  end  of  the  route.  When  an  entity  arrives  at  the  last  route 
waypoint,  new  capabilities  allow  the  user  to  choose  to  command  the  assigned  entity  to  either 
automatically  loop  back  around  to  the  beginning  waypoint  and  repeat  the  maneuver  or  to  turn 
around  180  degrees  and  return  on  the  path  in  the  reverse  direction.  Users  may  now  command  an 
XCITE  entity  to  repeat  these  actions  a  certain  number  of  times  before  performing  any  subsequent 
tactics  or  maneuvers.  For  land  vehicles  driving  a  route,  or  even  for  lifeforms  walking  along  a 
predefined  path,  the  scenario  provider  may  also  choose  to  have  the  assigned  player  entity  slow 
down  and  stop  briefly  at  each  waypoint. 

2.3.2. 19  Datalink  Setup  and  Control 

Further  research  and  development  is  needed  to  improve  the  way  scenario  providers  in  a  DMO 
environment  will  setup,  control,  and  use  simulated  radio  datalinks.  Existing  user  interfaces,  while 
useful  for  integration  checkout  and  test,  should  be  made  easier  to  operate  when  using  XCITE 
during  training  exercises.  Instructors  and  operators  want  to  see  and  feel  datalink  displays,  which 
are  similar  to  panels  and  screens  used  in  the  aircraft  and  by  ground  personnel.  They  are  already 
familiar  with  working  systems  like  those  they  use  in  the  field,  and  they  often  expect  the  training 
environment  will  employ  similar  tools  and  interfaces.  AFRL’s  government-owned  software 
applications  should  also  be  modified  so  that  repetitive  and  tedious  actions  may  be  minimized 
where  possible  when  using  XCITE  to  train  datalinks. 

Further  development  is  recommended  to  give  XCITE  datalink  software  routines  the  ability  to 
automatically  generate  surveillance  track  numbers.  Additionally,  effort  should  be  applied  to 
make  it  easier  for  scenario  providers  to  search  for  track  numbers  in  a  DMO  training  event  and 
visualize  the  datalink  picture. 

2.3.2.20  Chaff  Timeout 

The  WRSTP  team  modified  the  software  source  code  in  XCITE  to  cause  chaff  entities  to  be 
removed  from  the  simulated  scenario  following  a  period  of  time.  Previously,  the  entities 
remained  for  an  unnecessarily  longer  period,  which  tied  up  the  computer’s  memory  space  and 
made  it  unavailable  for  use  with  other,  more  current  entities. 

2.3.2.21  Optical  Sensors 

Simulated  human  vision  and  the  ability  of  XCITE  lifeform  entities  to  detect  and  track  moving  or 
stationary  objects  is  research  and  development  work  that  should  continue.  The  WRSTP  team  has 
observed  an  inadequate  manner  or  ability  in  which  XCITE  performs  the  functions  of  optical 
sensor  target  acquisition  and  tracking,  and  explored  possible  solutions.  However,  insufficient 
time  was  available  to  complete  this  study  under  TO  0037  or  to  propose  a  working  solution. 
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2.3.2.22  DIS  Output  Buffer  Size 

XCITE  has  an  output  buffer  that  can  hold  300  DIS  packets.  If  a  query  request  comes  in  for  a  list 
of  all  player  models,  then  XCITE  will  push  its  response  into  that  buffer  and  this  takes  up  more 
than  a  third  of  the  available  space.  While  the  XCITE  NIU  is  pushing  out  the  packets,  the 
application  may  receive  another  query  request.  Its  response  will  be  added  to  the  queue,  whether 
it  is  one  packet  of  tactical  area  coordinates,  125  packets  of  additional  player  model  infonnation 
such  as  jammer  capabilities  or  passive  detection  parameters,  or  a  packet  containing  a  response  to 
a  request  for  only  64  bits  of  infonnation  for  a  single  player  entity. 

If  requests  come  in  faster  than  XCITE  can  respond  then  the  console  will  print  messages 
indicating  an  overflow  error  condition  and  the  requesting  application  will  not  get  the  expected 
response.  The  WRSTP  team  explored  this  and  observed  the  Threat  IOS  display,  dialogs,  and 
menus  were  getting  stale  because  XCITE  was  busy  or  overloaded  with  requests  for  information. 
They  developed  a  modification  to  address  this  by  throttling  the  queries  that  Threat  IOS  sends  to 
XCITE.  It  waits  to  receive  all  packets  for  each  request  before  sending  the  next  request  to  XCITE. 
Potentially,  any  other  application,  which  communicates  with  XCITE,  may  still  experience 
problems  unless  it  does  something  to  slow  the  flow.  The  modification  puts  almost  all  requests  for 
information  into  a  queue.  This  queue  would  be  controlled  inside  the  Threat  IOS  to  limit  the 
possibility  of  XCITE ’s  output  buffer  filling  up. 

2.3.2.23  Coordinate  System  Selection  and  Resolution 

Scenario  providers  in  the  operational  training  community  who  use  the  government-owned 
XCITE  program  were  critical  of  the  fact  that  the  manual  task  of  creating  a  constructive  entity 
offered  too  few  options  for  defining  the  player’s  initial  position.  There  existed  a  few  different 
means  of  choosing  or  entering  a  location  developed  previously  under  other  task  orders.  The 
Mesa  Research  Site  further  developed  a  customizable  user  interface  for  selecting  the  coordinate 
system  used  by  the  Threat  IOS  application.  The  new  options  include  universal  transverse 
mercator  (UTM)  and  military  grid  reference  system  (MGRS),  in  addition  to  the  legacy  Lat/Long 
format.  Also,  the  operator  may  customize  the  Lat/Long  entry  to  be  provided  in  either 
DD:MM:SS  or  DD:MM.MMMM,  and  may  further  choose  the  resolution  of  the  decimal  minutes 
entry  to  be  any  number  of  decimal  points  from  one  to  six.  R&D  work  that  began  but  did  not 
complete  before  the  closure  of  this  task  included  modifications  to  another  Threat  IOS  dialog  that 
accepted  user  inputs  for  target  position  when  setting  up  for  commanding  fire  missions.  End-users 
want  to  be  able  to  enter  target  coordinates  relative  to  the  position  of  friendly  ground  forces,  in 
addition  to  the  currently  available  option  of  simply  entering  latitude  and  longitude.  We 
recommend  developing,  testing,  and  fielding  and  applying  the  new  format  and  resolution  options 
described  above  to  other  existing  Threat  IOS  user  dialogs  that  accept  coordinate  inputs. 
Additionally,  adjustments  should  be  made  to  the  application  software  so  that  the  cursor  can  be 
moved  automatically  from  one  dialog  entry  field  to  the  next  when  the  user  presses  the  tab  key 
while  entering  coordinates  and  other  parameters. 

We  recommend  enhancing  AFRL’s  XCITE  and  Threat  IOS  computer  program  software  to 
support  Global  Area  Reference  System  (GARS).  GARS  is  used  to  rapidly  and  clearly  define 
geographical  locations  for  battle-space  coordination,  deconfliction,  and  synchronization  as  well 
as  for  large-area  search-and-rescue  efforts.  Unlike  the  geographical  grid  systems  used  for  the 
different  operational  areas,  GARS  is  global  in  scope,  providing  a  common  language  between  the 
services/combatant  commands  and  simplifying  communications.  GARS  provides  a  common 
language  to  describe  30  x  30,  15  x  15,  and  5x5  minute  areas  unless  the  combatant  commander 
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determines  that  the  use  of  another  area  reference  system  is  mission  critical.  It  provides  the  two- 
dimensional  (2D)  construction  from  which  control  and  coordination  measures  can  be 
constructed.  It  is  not  used  to  describe  exact  geographic  locations  or  to  express  precise  positions 
for  guided  weapon  employment,  or  to  describe  areas  smaller  than  5  minutes  by  5  minutes. 

2.3.2.24  Weapon  Load  Outs 

CGF  applications  including  XCITE  and  Threat  IOS  should  undergo  further  development  to  give 
end  users  a  simple,  intuitive  way  to  adjust  the  weapon  load  out  of  an  entity  that  has  already  been 
created  in  the  DMO  environment.  Alternatively,  one  in  which  the  user  wants  to  create  but  with  a 
different  weapons  load  than  the  default  settings  that  are  stored  somewhere  in  a  database. 

2.3.2.25  Inflight  Weapon  Status 

WRSTP  engineers  explored  what  would  be  necessary  to  enhance  the  Threat  IOS  application  to 
present  to  operators  detailed  up-to-date  status  of  inflight  weapons  that  are  created  by  a  virtual 
cockpit  operation  in  a  DMO  event.  Recommended  work  includes  enhancements  to  the  network 
interfaces  of  cockpit  devices  in  order  to  publish  special  purpose  DIS  data  PDUs.  The  infonnation 
contained  in  each  packet  would  then  have  to  be  parsed  by  the  Threat  IOS  and  be  stored  in  local 
memory.  The  graphical  presentation  on  the  Threat  IOS  should  be  enhanced  to  allow  the  user  to 
browse,  choose,  and  review  weapon  modes,  electronic  warfare  emissions,  jamming 
effectiveness,  and  energy. 

2.3.2.26  Cognitive  Study  of  the  XCITE  GUI 

The  WRSTP  team  enhanced  the  XCITE  GUI  to  reduce  button  selections  used  to  turn  on  and  off 
various  symbols,  icons,  and  readouts  shown  on  the  Threat  IOS  display.  The  buttons  had  been 
arrayed  in  a  static  layout  that  gave  users  no  option  to  rearrange  their  appearance.  This  resulted  in 
an  inordinate  amount  of  time  required  to  locate  any  given  button.  The  modification  we 
introduced  makes  buttons  user-customizable  to  a  large  degree.  Future  growth  of  DMO  training 
devices  similar  to  and  including  the  Threat  IOS  application  should  be  designed  or  redesigned, 
where  practical,  to  give  end-users  more  control  over  the  appearance  of  their  user  interfaces  and 
controls.  This  will  further  reduce  tedium  and  operator  workload. 

2.3.2.27  Symbology 

The  WRSTP  team  observed  that  end-users  and  research  scientists  want  unique  symbols 
displayed  on  a  CGF  IOS  for  representing  simulated  entities,  which  may  be  relatively  similar  in 
many  regards  except  for  instance,  country  of  manufacture  or  antenna  height.  Our  engineers 
modified  Threat  IOS  to  give  users  a  way  that  they  can  individually  define  the  symbol  and 
associated  descriptive  text  for  all  known  (and  unknown)  types  of  sensors  and  emitters  defined  in 
Simulated  Interoperability  Standards  Organization  and  Combat  Air  Force  (CAF)  DMO 
standards.  If  users  want  more  ability  to  define  appearance  color  or  font  type  and  size  then  work 
should  continue  to  enhance  Threat  IOS  symbology  software  and  data  tables. 

2.3.2.28  User  Preferences 

Multiple  users  assigned  to  use  the  same  GUI  to  conduct  DMO  training  events  that  occur  at 
different  times  and  across  several  work  shifts  and  workdays  are  users  that  are  seeking  a  solution 
to  an  ongoing  problem  with  setting  personal  preferences.  The  WRSTP  team  observed  that 
research  scientists  and  end-users  wanted  to  set  their  own  user  preferences  with  the  XCITE  CGF 
and  Threat  IOS  GUI.  Additionally,  they  did  not  want  to  be  forced  to  use  other  user’s  preferences 
or  have  to  restore  settings  when  they  were  finished  with  their  work  shift.  A  low-risk,  low-cost 
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solution  to  this  problem  is  very  possible  but  it  has  not  been  implemented  due  to  other  higher 
priority  taskings. 

2.3.2.29  Areas  and  Boundaries 

WRSTP  engineers  enhanced  the  XCITE  CGF  to  make  use  of  user-defined  areas  and  boundaries 
in  the  employment  of  player  tactics  and  maneuvers.  Unfinished  work  includes  additional  changes 
to  easily  create  and  edit  the  associated  data  and  to  view  them  on  the  Threat  IOS  display. 

2.3.2.30  Joint  Precision  Airdrop  System  (JPADS) 

Four  members  of  the  C-17  training  community  met  with  AFRL  government  personnel  and  four 
WRSTP  XCITE  development  team  members.  Visiting  AFRL  was  John  W.  Hagin,  USAF  AFMC 
HQ  AFMC/A4VQ,  Rob  Gomez,  C-17  Program  Manager,  Maj  Paul  Adams,  and  TSgt  Joshua 
Hoekstra.  The  topic  of  the  meeting  was  training  requirements  involving  weather  measurements 
transmitted  by  the  radiosondes  the  C-17  loadmaster  drops  prior  to  when  the  aircrew  lands  or 
prior  to  dropping  cargo  palettes.  We  also  discussed  simulating  the  guidance  mechanism  mounted 
atop  the  cargo  palette.  The  visitors  described  preflight  tasks  and  sortie  operations,  which  they 
need  to  provide  training.  The  C-17  community  explored  use  of  AFRL’s  XCITE  application  to 
provide  the  input  to  their  laptop  computer  on  an  Ethernet  port.  The  visitors  stated  that  there  is  no 
cost  effective  simulation  that  can  train  for  these  tasks.  Training  currently  occurs  during  sorties  at 
Ft  Sill  and  does  not  provide  complete  training  of  all  tasks. 

2.3.2.31  Entity  Pairing  Status 

End-users  and  research  scientists  require  status  and  feedback  about  the  interactions  between  pairs 
of  entities  engaged  with  one  another  in  a  DMO  training  event.  Using  XCITE  CGFs,  they 
observed  sensor  mode  changes  but  could  not  easily  or  accurately  ascertain  the  reason  or  cause 
for  the  mode  change.  Likewise,  weapon  engagements  did  not  previously  occur  with  the  requisite 
status  infonnation  displayed  on  a  screen  for  users  to  determine  why  a  weapon  did  not  fire  or  why 
a  different  weapon  fired  than  expected.  In  the  XCITE  design,  these  conditions  are  known  by  the 
algorithms  that  process  sensors  and  weapons.  Previously,  the  information  was  not  published  or 
displayed.  The  WRSTP  team  enhanced  the  XCITE  NIU  and  Threat  IOS  to  display  more 
information  to  help  the  user  understand  and  know  what  is  happening  in  the  decision  logic  within 
XCITE  while  a  scenario  is  playing  out.  Users  should  be  solicited  about  the  type  of  infonnation 
that  they  want  to  see  displayed  during  training  events. 

2.3.2.32  Classification  Level 

The  WRSTP  engineers  enhanced  XCITE  to  add  a  security  measure  that  checks  the  classification 
levels  of  scenario  and  model  data  files  to  make  sure  they  match  the  user’s  intent  when  preparing 
to  begin  a  training  mission.  Previously,  it  was  possible  to  mistakenly  initialize  a  classified 
mission  even  if  the  application  was  in  an  unclassified  mode  of  operation. 

2.3.2.33  Database  Schema  and  Forms 

Previously,  AFRL’s  XCITE  Access  database  was  a  binary  object.  Changes  customarily  made  by 
any  engineer  to  the  XCITE  Access  database  were  not  directly  visible  to  other  developers.  There 
had  been  no  easy  means  to  visualize  independent  changes  of  individual  modules.  WRSTP 
engineers  created  a  new  Access  database  module  that  extracts  all  data  objects  into  text  files  that 
are  human  readable  and  that  can  be  imported  back  into  the  database.  This  facilitates 
configuration  management  (CM)  by  allowing  the  CM  system  to  track  schema,  form,  report,  and 
even  data  changes  at  a  text  level  rather  than  tracking  alterations  to  a  binary  object. 
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2.3.2.34  XCITE  Training  Classes 

Over  the  course  of  the  task  order,  the  WRSTP  team  has  had  several  opportunities  to  train  visitors 
how  to  use  and  collaboratively  develop  the  XCITE  application.  Trainees  benefited  from  learning 
first-hand  from  experienced  research  and  development  engineers  how  to  effectively  apply  the 
capabilities  of  the  tool,  contribute  infonnation  to  the  database,  and  enhance  the  computer 
algorithms.  In  addition  to  the  documentation  and  training  videos  prepared,  the  user  community 
and  co-developers  gained  insight  into  the  inner  workings  of  XCITE  and  gained  knowledge  of 
how  to  further  develop  the  software  suite  for  their  unique  needs  and  requirements. 

2.3.2.35  Electronic  Warfare/Electronic  Attack  Debrief  Capability 

The  WRSTP  team  built  a  questionnaire  in  Excel  that  received  the  research  psychologist’s 
approval.  The  psychologists  were  impressed  the  knowledge  of  research  techniques  and  concerns, 
and  made  only  minor  changes — mostly  in  the  demographics  area.  The  team  decided  to  use 
Access  vs.  Excel,  as  Access  is  a  more  powerful  program  in  this  application.  AFRL  government 
personnel  received  the  approval  of  the  Institutional  Research  Board  (IRB)  to  use  the 
questionnaire.  The  questionnaire  was  never  administered  because  of  the  lack  of  test  subjects  due 
to  the  schoolhouse  closing  down. 


2.4  Audio  Integration/Demonstrations 

The  application  listen  renders  audio  for  entities  as  well  as  detonation  and  fire  events  in  three 
spatial  dimensions  relative  to  a  listener.  The  listen  software  was  government  furnished,  provided 
by  71 1  HPW/RHCB.  The  WRSTP  team  demonstrated  the  feasibility  of  integrating  listen  spatial 
audio  into  the  Joint  Tenninal  Attack  Controller  (JTAC)  Training  and  Rehearsal  System  (TRS), 
which  includes  a  3D  environmental  sound  capability  to  increase  immersion  in  the  simulated 
battlefield. 

2.4.1  Methods,  Assumptions,  and  Procedures 

Examination  of  listen  source  code  revealed  that  it  could  not  be  used  as-is.  While  it  provided  the 
functionality  desired  by  the  team,  it  relied  heavily  on  the  VR-Link  networking  toolkit  libraries 
developed  by  MaK  Technologies.  Use  of  these  libraries  would  have  required  the  purchase  of 
licenses  from  MaK.  The  cost  -  exceeding  $10,000  for  a  one-time  purchase  -  rendered  this  option 
impractical. 

Listen  is  an  application  that  mainly  processes  entity  state  updates  and  fire/detonation  events,  and 
sends  the  AudioServer  the  necessary  data  regarding  the  location  of  each  potential  sound  source 
relative  to  the  listener.  We  decided  to  start  from  an  unclassified  version  of  the  host  used  by  the 
Radio  Frequency  Threat  Environment  (RFTE).  This  host  (hereinafter  referred  to  as  the  Spatial 
Audio  Host),  was  originally  designed  as  a  passive  application  that  listened  to  distributed 
interactive  simulation  network  (DISNET)  and  processed  entity  states  and  other  distributed 
interactive  simulation  (DIS)  events  including  fire  and  detonation  events. 

The  approach  decided  upon  was  to  remove  the  functionality  related  to  text-to-speech  (which 
JTAC  does  not  use),  and  add  the  necessary  Audio  Server/slab  3d  calls  to  the  application,  using 
listen  as  a  guide  for  updating  entity  states  and  handling  fire/detonation  events  vis-a-vis  the 
AudioServer  library  calls.  Additionally,  since  a  different  head  tracker  was  used  on  site  than  at 
Wright-Patterson  Air  Force  Base,  code  would  be  added  to  read  the  tracker  data  broadcast  by  the 
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JTAC  observer’s  compass  to  provide  the  necessary  listener  orientation  information  to  the 
AudioServer;  trackDAPI  would  not  be  used. 

2.4.2  Results  and  Discussion 

The  WRSTP  team  integrated  and  demonstrated  existing  711  HPW/RHC-provided  audio  tools 
into  AFRL/RHA  simulation  systems,  specifically  the  JTAC-TRS.  We  provided  technical 
information  that  supported  a  common  architecture  definition  and  system  and  software 
development,  with  consideration  of  DIS/high  level  architecture  (HLA)  protocols,  of  a  single  set 
of  improved  spatial  (3D)  audio  tools  that  can  be  used  to  maintain  multiple  types  of  voice 
communication  on  a  network  (voice,  chat,  speech-to-text,  text-to-speech,  speaker  location,  etc.). 

2.5  Ultra-High  Resolution  Deployable  Projector  Components  for  Simulation 

Initially  this  effort  focused  on  developing  and  testing  optical  components  for  micro-electro- 
mechanical  based  laser  projectors  for  immersive  environments.  The  focus  shifted  to  investigating 
and  developing  specifications  and  standards  for  the  acquisition  of  visual  systems  for  simulation 
(Immersive  Display  Evaluation  and  Assessment  Study  [IDEAS]). 

2.5.1  Methods,  Assumptions,  and  Procedures 

Work  began  by  collaborating  with  Alces  Technology  to  integrate  display  technology  components 
into  existing  government  furnished  projectors  at  the  Mesa  Research  Site.  Based  on  this 
integration,  the  plan  was  to  define  image  generation  requirements,  purchase  and/or  develop 
required  hardware  and  software,  and  interface  the  projectors  to  their  respective  image  generation 
systems.  Upon  completion  of  the  system  integration,  human  perceptual  and  engineering  tests  to 
characterize  the  functional  performance  of  the  display  systems  were  to  be  conducted. 

In  late  2009,  AFRL  and  Aeronautical  Systems  Center  (ASC)  proposed  and  initiated  Phase  I  of 
IDEAS,  which  launched  in  the  summer  of  2010.  The  following  are  some  of  the  long-tenn  goals 
of  this  program: 

•  Develop  display  system  performance  requirements  for  Air  Force  programs  that  are  defensible 
based  on  training  task  performance. 

•  Define  metrics  and  methods  of  measuring  display  systems  to  assess  conformance  with  the 
requirements. 

•  Define  the  requirements  for  a  decision  support  system  (DSS)  that  would  provide  stakeholders 
ready  access  to  the  requirements,  metrics,  and  measurement  procedures. 

•  Demonstrate  the  utility  of  the  approach  so  that  long  tenn  funding  might  be  secured  to  sustain 
the  extension  and  maintenance  of  the  requirements  and  DSS. 

•  Monitor  and  facilitate  the  development  of  the  DSS  by  a  third  party  and  feed  the  developer 
content  to  extend  the  system. 

For  the  first  phase  of  the  IDEAS  program,  the  scope  was  limited  to  fast  jet  training  of  daylight 
operations  with  an  emphasis  on  the  F-16  Mission  Training  Center  (MTC).  Discussions  with  fast 
jet  subject  matter  experts  (SMEs)  resulted  in  selecting  the  aircraft  visual  identification  task  for 
the  first  modeling  effort.  Discussions  with  key  Air  Force  acquisition  professionals  produced  a  list 
of  display  design  parameters  that  were  in  need  of  development  or  rework.  High  on  the  list  of 
priorities  was  the  need  for  a  requirement  and  metric  that  regulated  the  motion  induced  blurring. 
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The  team  focused  its  efforts  on  the  effects  of  five  variables  expected  to  be  the  primary 
detenninants  of  task  performance  with  moving  images  on  high-resolution  display  systems.  These 
five  variables  included:  angular  velocity  of  the  image,  pixel  hold  time,  pixel  pitch,  display 
luminance,  and  display  contrast. 

During  the  fall  of  2010,  a  model  of  the  effects  of  these  and  related  variables  on  aircraft 
identification  range  was  prepared  and  initially  tested  using  the  data  from  the  few  evaluations 
found  in  the  literature.  The  paper  presented  at  the  2011  IMAGE  Conference  (Lloyd,  Williams, 
and  Pierce,  2011)  describes  the  model  and  initial  validation. 

Over  the  next  few  months,  the  team  composed  and  conducted  two  laboratory  evaluations  to 
collect  a  much  larger  set  of  data  to  validate  and  tune  the  model.  Across  these  two  evaluations,  we 
measured  aircraft  identification  range  as  a  function  of  420  combinations  of  the  settings  of  five 
design  variables:  pixel  hold  time,  angular  velocity,  pixel  pitch,  display  system  luminance,  and 
display  system  contrast. 

2.5.2  Results  and  Discussion 

The  collaboration  with  Alces  Technology  Inc,  to  develop  and  test  optical  components  for  micro- 
electro-mechanical  based  laser  projectors  was  suspended  by  mutual  agreement  between  Alces 
and  the  Government.  This  collaborative  work  initially  consisted  of  upgrading  AFRL’s  Evans  and 
Sutherland  Laser  Projector  (ESLP)  with  new  optical  and  Micro-Electrical-Mechanical  Systems 
(MEMS)  components  developed  by  Alces.  As  technology  development  continued,  it  became 
apparent  that  these  new  components  were  incompatible  with  the  ESLP.  This  led  to  the 
suspension  of  any  direct  collaboration  between  Alces  and  L-3.  The  government  redirected  the  L- 
3  to  focus  on  developing  the  human  perceptual  and  engineering  tests  used  to  characterize  the 
functional  performance  of  displays,  such  that  these  tests  will  be  fully  developed  to  employ  on 
next  generation  display  systems,  including  the  display  technologies  being  developed  by  Alces 
Technology.  This  development  of  functional  performance  metrics  is  the  IDEAS  research  effort. 
The  WRSTP  team  researched  and  developed  methods  for  evaluating  the  fidelity  of  immersive 
environment  display  systems. 

The  IDEAS  program  was  initiated  in  the  summer  of  2010.  The  long-term  goal  of  this  program  is 
to  produce  a  DSS  that  will  (1)  enable  Air  Force  acquisition  professionals  to  rapidly  generate 
defensible  display  system  requirements  given  a  set  of  training  task  requirements,  (2)  enable 
suppliers  to  develop  new  visual  systems  and  prepare  proposals,  and  (3)  aid  certifiers  to  measure 
perfonnance.  Some  overarching  goals  for  the  DSS  include  the  following: 

•  Usable  by  persons  who  understand  the  training  need  but  are  not  experts  in  current  display 
technology 

•  Produces  defensible  requirements  based  on  task  performance  data  and  previously  validated 
models 

•  Identifies  the  data  and  models  that  underlie  each  recommendation 

•  Allows  high  level  design  trades  to  be  made  during  the  requirements  generation  process 

•  Produces  the  specific  text  and  levels  for  selected  display  system  requirements 

•  Provides  the  approved  measurement  procedure  for  each  requirement 

•  Employs  an  architecture  that  allows  updating  the  data  and  models  as  technologies  evolve 

Development  and  validation  of  the  first  model/data  to  be  incorporated  into  the  DSS  began  in  the 
fall  of  2010.  This  work  is  summarized  in  technical  reports  and  two  papers  presented  at  IMAGE 
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2011.  The  papers  and  presentations  describe  the  general  capabilities  of  the  DSS  and  provide 
concrete  examples  of  the  following: 

•  Generation  of  selected  display  requirements  from  the  F-16  MTC  training  task  list 

•  Use  of  the  DSS  to  make  source  selection  decisions 

•  Generation  of  measurement  procedures 

•  Use  of  the  DSS  by  a  supplier  for  product  development  planning 

Two  evaluations,  conducted  at  the  Mesa  Research  Site  using  F-16  subject  matter  experts  as 
participants,  were  used  to  test,  validate,  and  tune  the  model. 

3.0  CONCLUSIONS 

This  final  report  summarizes  work  completed  in  support  of  Task  Order  0037,  Entity  Modeling 
and  Immersive  Decision  Environments  research  and  development.  More  in-depth  documentation 
is  available  by  reviewing  the  technical  reports  and  presentation  materials  cited  in  the  Annotated 
Bibliography. 
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Question  3  Question  4  Question  5 

Question  6 

Question  7  Question  8 

Question  9  '  Question  10  Question  * 

Electronic  Warfare  Debrief 
Requirements  Survey 

Warfighter  Readiness  Research  Division 
Air  Force  Research  Laboratory 
Human  Effectiveness  Division 

6030  Kent  Street 
Mesa,  AZ  85206 


Following  are  20  questions  regarding  Electronic  Warfare  training  programs,  in  an 
attempt  to  gather  requirements  for  an  upgraded  debriefing  system.  You  will  find 
directions  on  every  tab  regarding  how  to  answer  the  question  on  that  tab. 

At  the  bottom  of  every  tab  you  will  find  a  Save  button.  Please  select  your 
answer(s)  for  that  tab,  then  click  the  Save  button.  If  you  move  to  a  different  tab 
without  Saving  on  the  current  tab  your  answers  will  be  lost  (including  this  tab). 
EVERY  TIME  you  click  the  Save  button  any  answers  already  on  that  tab  will  be 
replaced  by  the  new  answers,  and  you  will  be  auto-forwarded  to  the  next  tab.  The 
blue  arrows  at  the  upper  right  allow  for  navigation  to  further  tabs. 

Please  fill  in  the  Demographic  information  requested  below,  and  click  Save  at  the 
bottom  of  the  tab.  You  will  be  auto-forwarded  to  Question  1. 


Respondent  ID: 
Flight  Hours: 
Instructor: 
Instructor  Hours: 

Evaluator: 
Evaluator  Hours: 
EW  Focus: 
Crew  Position: 


■ 


■ 


Airframes  In 
Which  You 
Have  Been  An 
Instructor 
And/Or 
Evaluator: 
(Hold  CTRL 
and  click  on 
all  that  apply) 


Save 


A-10 

AC-130H 

AC-130U 

B-l 

B-2 

B-52 

C-130 

C-17 

E-3 

E-4 

E-8 

EA-6B 

EC-130H 

EC-130J 

EC-135 


□ 


The  Privacy  Act  of  1974  applies  to  all  information  gathered  in  this  survey 


Figure  A.1.  Demographics 
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Demographics  Question  1  Question  2  |  Question  3  Question  4  Question  5  |  Question  6  Question  7  Question  8  Question  9  |  Question  10  j  Question  '  ’ 

What  do  you  consider  to  be  the  strengths  in  your  current  Electronic  Warfare/Electronic  Attack  (EW/EA) 
training  program?  (Hold  CTRL  and  select  all  that  apply) 


Classroom  Instruction 
Doctrinal  Study 
Group  Study  Time 

Integrated  Air  Defense  System  Study 
Live  Flight  Time  Against  Threat  Simulators 
Simulator  Time 
Solo  Study  Time 
System  "Switchology" 

Tactics,  Techniques,  and  Procedures  (TTPs)  Study 
Threat  System  Recognition 

Other  (please  use  the  space  below  to  add/elaborate) 


Other: 


Figure  A.2.  Question  1 
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Demographics  |  Question  1  Question  2  ^Question  B  j  Question  4  j  Question  5  j  Question  6  ;  Question  7  ;  Question  8  Question  9  Question  10  |  Question  '  j 

What  do  you  consider  to  be  the  weaknesses  in  your  current  EW/EA  training  program?  (Hold  CTRL  and 
select  all  that  apply) 


Classroom  Instruction 
Doctrinal  Study 
Group  Study  Time 

Integrated  Air  Defense  System  Study 
Live  Flight  Time  Against  Threat  Simulators 
Simulator  Time 
Solo  Study  Time 
System  "Switchology" 

Tactics,  Techniques,  and  Procedures  (TTPs)  Study 
Threat  System  Recognition 

Other  (please  use  the  space  below  to  add/elaborate) 


Other: 


Save 


Figure  A.3.  Question  2 


23 

DISTRIBUTION  A.  Approved  for  public  release;  distribution  unlimited.  (Approval  given  by 
88  ABW/PA,  88ABW-2012-1090,  5  Mar  12.) 


Demographics  j  Question  1  |  Question  2 1  Question  3  j  Question  4  Question  5  |  Question  6  |  Question  7  |  Question  8  |  Question  9  |  Question  10  j  Question  ‘  | 

Accepting  as  granted  that  all  of  the  following  concepts  are  important  for  a  trainee  to  take  away  from 
EW/EA  training,  please  select  your  "top  10"  in  priority  order. 


Other: 


Save 


Figure  A.4.  Question  3 


24 

DISTRIBUTION  A.  Approved  for  public  release;  distribution  unlimited.  (Approval  given  by 
88  ABW/PA,  88ABW-2012-1090,  5  Mar  12.) 


Demographics  |  Question  1 1  Question  2  |  Question  3  |  Question  4  |  Question  5  |  Question  6  Question  7  [  Question  8  Question  9  |  Question  10  |  Question  1  1  > 

QUESTION  REMOVED.  DO  NOT  SAVE,  PROCEED  TO  NEXT  TAB. 


Figure  A.5.  Question  4 
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Demographics  [  Question  1  !  Question  2  |  Question  3  Question  4  Question  5  [  Question  6  j  Question  7  |  Question  8  Question  9  |  Question  10  j  Question  1  |  > 

Which  of  the  following  information  sets  you  believe  would  be  important  in  an  EW/EA  event  debrief? 

(Select  your  "top  10"  in  priority  order) 


Priority  1: 
Priority  2: 
Priority  3: 
Priority  4: 
Priority  5: 
Priority  6: 
Priority  7: 
Priority  8: 
Priority  9: 
Priority  10: 


Other: 


Save 


Figure  A.6.  Question  5 


0 
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Demographics 


Other: 


|  Question  2 

Question  3  | 

|  Question  4 

Question  5 

Question  6 

What  would  be  the  optimal  flow  of  presentation  for  the  debrief?  (Select  in  priority  order) 


Priority  1: 
Priority  2: 
Priority  3: 
Priority  4: 


Save 


Figure  A.7.  Question  6 
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Demographics 


Other: 


Question  1  |  Question  2  |  Question  3  j  Question  4  Question  5  |  Question  6  I  Question  7  I  Question  8  j  Question  9  [  Question  10  |  Question  * 

What  would  be  the  easiest  way  to  present/interpret  debrief  data?  (Select  in  priority  order) 


Priority 

Priority 

Priority 

Priority 


□ 

3 

□ 

a 

Save 


Figure  A.8.  Question  7 
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Demographics  |  Question  1  I  Question  2  |  Question  3  j  Question  4  Question  5  |  Question  6  j  Question  7  Question  8  |  Question  9  |  Question  10  ;  Question  1  |  > 

In  terms  of  visualization  during  debrief,  estimate  the  potential  effectiveness  of  the  following  visual 
depictions.  (Select  in  priority  order) 


Priority  1: 
Priority  2: 
Priority  3: 
Priority  4: 
Priority  5: 
Priority  6: 
Priority  7: 
Priority  8: 
Priority  9: 
Priority  10: 


Other: 


Save 


Figure  A.9.  Question  8 


0 
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Demographics 

Question  1 

Question  2  Question  3 

Question  4 

Question  5 

Question  6  Question  7 

Question  8 

Question9 

Question  10 

Question 

In  terms  of  training  effectiveness,  estimate  the  potential  effectiveness  of  viewing  a  threat  engagement 
sequence  from  start  to  finish  from  the  viewpoint  of  the  threat.  (Select  one,  and  feel  free  to  elaborate  in 
the  space  below) 


Definite  Positive  Effect 
Probable  Positive  Effect 
Possible  Positive  Effect 
No  Potential  Effect 
Possible  Negative  Effect 
Probable  Negative  Effect 
Definite  Negative  Effect 


Other: 


Save 


Figure  A.10.  Question  9 
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Demographics  Question  1  Question  2  )  Question  3  j  Question  4  j  Question  5  |  Question  6  |  Question  7  |  Question  8  |  Question  9  Question  10  Question  ' 

In  terms  of  training  effectiveness,  estimate  the  potential  effectiveness  of  having  a  side-by-side  view  of  the 
engagement  sequence-from  start  to  finish-with  all  pertinent  sensors'  displays  from  both  Friendly  and 
Hostile.  (Select  one,  and  feel  free  to  elaborate  in  the  space  below) 


Definite  Positive  Effect 
Probable  Positive  Effect 
Possible  Positive  Effect 
No  Potential  Effect 
Possible  Negative  Effect 
Probable  Negative  Effect 
Definite  Negative  Effect 


Other: 


Save 


Figure  A.11.  Question  10 


31 

DISTRIBUTION  A.  Approved  for  public  release;  distribution  unlimited.  (Approval  given  by 
88  ABW/PA,  88ABW-2012-1090,  5  Mar  12.) 


32 

DISTRIBUTION  A.  Approved  for  public  release;  distribution  unlimited.  (Approval  given  by 
88  ABW/PA,  88ABW-2012-1090,  5  Mar  12.) 


Question  2  Question  3  Question  4  Question  5  Question  6  Question  7  j  Question  8  Question  9  Question  10  j  Question  11  Question  12  Question  1  ' 

If  displaying  Pk  calculations  would  have  a  positive  effect,  estimate  which  would  have  the  biggest  positive 
effect.  (Hold  CTRL  and  select  all  that  apply) 


I  do  not  believe  seeing  the  actual  Pk  calculations  would  have  a  positive  effect 

Electronic  Countermeasures  Effects  Calculations 

Expendable  Countermeasure  vs.  Seeker  Head  Calculations 

Expendable  Countermeasure  vs.  Threat  Radar  Calculations 

Expendables/Maneuver  vs.  Seeker  Head  Calculations 

Expendables/Maneuver  vs.  Threat  Radar  Calculations 

Maneuver  vs.  Seeker  Head  Calculations 

Maneuver  vs.  Threat  Radar  Calculations 

Threat  Damage  to  Own-ship  Calculations 

Threat  Radar  Power  vs.  Jammer  Power  Calculations 

Total  Time  Within  Radar  Lock  (Missile  or  AAA) 

Other  (please  use  the  space  below  to  add/elaborate) 


Other: 


Save 


Figure  A.13.  Question  12 
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Question  3  1  Question  4  1  Question  5  1  Question  6  1  Question  7 

Question  8  Question  9 

Question  10  Question  11  Question  12  Question  13 

Question  ' 

In  terms  of  audio  information  during  debrief,  estimate  the  potential  effectiveness  of  the  following  audio 
depictions.  (Select  in  priority  order) 


Priority  1: 
Priority  2: 
Priority  3: 
Priority  4: 
Priority  5: 
Priority  6: 
Priority  7: 
Priority  8: 
Priority  9: 


Other: 


Save 


Figure  A.14.  Question  13 


a 


a 
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Question  4  Question  5  Question  6  Question  7  Question  8  Question  9 


Question  10  Question  11  Question  12  Question  13  Question  14  Questio 


Assuming  a  mission  profile  wherein  the  launch/recovery  base  is  located  outside  the  Operating  Area  (vice 
inside  the  AO,  e.g.  A-lOs  in  Afghanistan  during  OEF),  at  what  point  during  the  mission  would  it  be 
appropriate  to  start  collecting  data?  (Select  one) 


Engine  Start/Taxi/Takeoff 

Fence  Check/Operating  Area  Penetration 

On-Station 

First  Detection  by  Early  Warning  Radar 

First  Detection  by  Search/Acquisition  Radar 

Other  (please  use  the  space  below  to  add/elaborate) 


Other: 


Save 


Figure  A.15.  Question  14 
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Question  12  |  Question  13  Question  14  Question  15  Question  16  Question  17  Question  18  |  Question  19  Question  20  Question  21  |  End 

What  scenario  data  would  you  use  during  debrief  to  instruct  trainees  to  recognize  indications  of  adversary 

Electronic  Attack?  (Select  in  priority  order) 

Priority  1: 

3 

Priority  2: 

3 

Priority  3: 

3 

Priority  4: 

3 

Priority  5: 

3 

Priority  6: 

3 

Priority  7: 

3 

Priority  8: 

3 

Priority  9: 

3 

Other: 

Save 

Figure  A.16.  Question  15 
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Question  12  |  Question  13  |  Question  14  |  Question  15  j  Question  16  ,  Question  17  |  Question  18  |  Question  19  |  Question  20  |  Question  21 1  End 

In  terms  of  training  effectiveness,  estimate  the  potential  effectiveness  of  seeing  a  side-by-side  comparison 
of  a  trainee's  risk  management  plan  (e.g.  threat  avoidance,  SEAD  coverage,  etc.)  vs.  risk  as  derived  from 
AFTTP  3-1  (and/or  other  applicable  reference).  (Select  one,  and  feel  free  to  elaborate  in  the  space  below) 


Definite  Positive  Effect 
Probable  Positive  Effect 
Possible  Positive  Effect 
No  Potential  Effect 
Possible  Negative  Effect 
Probable  Negative  Effect 
Definite  Negative  Effect 


Other: 


Save 


Figure  A.17.  Question  16 
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Question  12  |  Question  13  Question  14  j  Question  15  |  Question  16  Question  17  Question  18  j  Question  19  Question  20  |  Question  21 1  End 

In  terms  of  training  effectiveness,  estimate  the  potential  effectiveness  of  being  able  to  dynamically  (i.e. 
"on  the  fly")  adjust  a  given  training  scenario  to  demonstrate  the  consequences  of  different  tactics  on  risk. 
(Select  one,  and  feel  free  to  elaborate  in  the  space  below) 


Definite  Positive  Effect 
Probable  Positive  Effect 
Possible  Positive  Effect 
No  Potential  Effect 
Possible  Negative  Effect 
Probable  Negative  Effect 
Definite  Negative  Effect 


Other: 


Save 


Figure  A.18.  Question  17 
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Question  12  |  Question  13  ]  Question  14  |  Question  15  |  Question  16  |  Question  17  Question  18  Question  19  |  Question  20  |  Question  21 1  End 

In  terms  of  training  effectiveness,  estimate  the  potential  effectiveness  of  having  a  "parallel"  scenario 
flown  to  AFTTP  3-1  (or  other  applicable)  standards,  such  that  trainees  could  have  a  "textbook"  mission  to 
compare  to  their  missions.  (Select  one,  and  feel  free  to  elaborate  in  the  space  below) 


Definite  Positive  Effect 
Probable  Positive  Effect 
Possible  Positive  Effect 
No  Potential  Effect 
Possible  Negative  Effect 
Probable  Negative  Effect 
Definite  Negative  Effect 


Other: 


Save 


Figure  A.19.  Question  18 
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Question  12 

Question  13  Question  14  |  Question  15  |  Question  16  Question  17 

Question  18 

Question  19 

Question  20  |  Question  21  |  End 

l  - 1  ► 

Without  consulting  Form  8  trend  data,  what  types  of  errors  are  most  common  among  trainees  in  your 
platform?  (Hold  CTRL  and  select  all  that  apply) 


Failure  to  "Capture"  a  Threat  (as  applicable) 

Failure  to  Geolocate  a  Threat 

Failure  to  Recognize  Adversary  Electronic  Attack 

Improper  Electronic  Jamming  Technique/Settings  (as  applicable) 

Improper  Employment  of  Expendable  Countermeasures  (as  applicable) 

Improper  Infrared  Jamming  Technique/Settings  (as  applicable) 

"Stepping  on"  Crew  Interphone  Comms 

"Stepping  on"  Radio  Comms 

"Switchology" 

Terminology 
Threat  Recognition 

Other  (please  use  the  space  below  to  add/elaborate) 


Other: 


Save 


Figure  A.20.  Question  19 
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Question  12  Question  13  Question  14  Question  15  Question  16  Question  17  Question  18  Question  19  Question  20  Question  21  End 

Of  the  list  of  errors  you  provided,  which  types  of  errors  are  most  hazardous  to  crew  and/or  mission 
completion?  (Select  in  priority  order) 


Priority  1: 
Priority  2: 
Priority  3: 
Priority  4: 
Priority  5: 
Priority  6: 
Priority  7: 
Priority  8: 
Priority  9: 
Priority  10: 


Other: 


Save 


- 

□ 

I 

□ 

□ 

- 

- 

- 

□ 

7 

Figure  A.21.  Question  20 
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Question  12  |  Question  13  Question  14  Question  15  Question  16  Question  17  Question  18  j  Question  19  Question  20  Question  21  End 

Imagine  an  "intelligent"  training  scenario  generation  capability,  such  that  a  training  device  could  identify 
trainee  weaknesses  over  time  and  suggest/build  subsequent  training  events  to  focus  on  those 
weaknesses.  Estimate  the  potential  effectiveness  of  such  a  system.  (Select  one,  and  feel  free  to  elaborate) 


Definite  Positive  Effect 
Probable  Positive  Effect 
Possible  Positive  Effect 
No  Potential  Effect 
Possible  Negative  Effect 
Probable  Negative  Effect 
Definite  Negative  Effect 


Other: 


Save 


Figure  A.22.  Question  21 
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Question  12  j  Question  13  '  Question  14  j  Question  15  |  Question  16  J  Question  17  |  Question  18  j  Question  19  j  Question  20  |  Question  21  j  End 


This  is  the  end  of  the  questionnaire.  If  you  would  like  to  go  back  and  re¬ 
read  or  change  an  answer,  feel  free  to  do  so.  Remember  to  click  the  save 
button  or  your  changes  will  be  lost. 

Please  notify  the  AFRL  personnel  when  you  are  done. 

Thank  you  for  taking  the  time  to  complete  this  survey.  Your  inputs  are  vital 
to  the  design  and  deployment  of  state-of-the-art  debriefing  technologies 
that  will  enable  future  aircrew  to  become  expert  aircrew  in  less  time. 


Figure  A.23.  End 
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ANNOTATED  BIBLIOGRAPHY 

Robust  Immersive  Decision  Environments  Research  (RIDER) 

Boydstun,  A.  S.,  Patterson,  R.,  Pierce,  B.,  Park,  L.  M.,  &  Shannan,  J.  (2010).  On  the 

Development  of  Training  Principles  for  Intuitive  Decision  Making.  Human  Factors:  The 
Journal  of  the  Human  Factors  and  Ergonomics  Society,  54(19),  1713-1716. 

Many  decisions  made  in  real-world  situations  involve  a  fonn  of  intuitive  pattern 
recognition.  One  way  to  investigate  training  principles  for  developing  this  type  of 
decision  making  utilizes  implicit  learning  in  an  immersive  environment,  where  training 
stimuli  are  generated  by  a  finite-state  algorithm.  In  the  current  study,  we  investigated  the 
effects  of  manipulating  training-sequence  length  and  algorithmic  complexity  in  an 
immersive  implicit-learning  paradigm.  Results:  training-sequence  length  interacted  with 
algorithmic  complexity  such  that  perfonnance  was  best  when  training-sequence  length 
was  long  and  the  algorithm  was  simple,  and  when  training-sequence  length  was  short  and 
the  algorithm  was  complex.  When  training  intuitive  decision  making,  training-sequence 
length  should  be  matched  to  algorithmic  complexity. 

Boydstun,  A.  S.,  Patterson,  R.,  Pierce,  B.,  Park,  L.  M.,  &  Tripp,  L.M.  (2011).  Articulatory 
Suppression  Affects  Situational  Pattern  Recognition  in  Immersive  Environments.  In 
Proceedings  of  the  10th  International  Conference  on  Naturalistic  Decision  Making, 
Orlando,  FL,  June. 

We  examined  several  factors  that  may  affect  training  for  intuitive  decision  making, 
namely  articulatory  suppression,  pattern  complexity,  and  stimulus  block  size.  Results: 
articulatory  suppression  and  pattern  complexity  impaired  decision  making,  but  block  size 
did  not.  Future  work  will  examine  how  attention  may  interact  with  training  for  intuitive 
decision  making. 

Patterson,  R.,  Fournier,  L.,  Pierce,  B.  J.,  Winterbottom,  M.  D.,  &  Tripp,  L.  M.  (2009). 

Modeling  the  Dynamics  of  Recognition-primed  Decision  Making.  In  Proceedings  of  the 
9th  Bi-annual  International  Conference  on  Naturalistic  Decision  Making  (NDM9)  23-26 
June  2009  at  the  British  Computer  Society,  Southampton  Street,  London,  UK. 

Two  types  of  decision-making  processes  have  been  identified  in  the  literature:  an 
analytical  process  and  an  intuitive  process.  One  conceptual  model  of  the  latter  is  the 
recognition-primed  decision  (RPD)  model.  According  to  this  model,  decision  making  in 
naturalistic  contexts  entails  a  situational  pattern-recognition  process  that,  if  subsequent 
expectancies  are  confirmed,  leads  the  decision  maker  to  render  a  decision  to  engage  in  a 
given  course  of  action.  In  this  paper,  we  describe  a  system  dynamics  model  of  Klein’s 
RPD  framework  that  focuses  upon  the  dynamics  of  the  decision-making  process.  The 
structure  of  our  RPD  model  is  based  on  a  model  of  a  set  of  laboratory  phenomena  called 
conjunction  benefits  and  costs,  which  was  extended  to  encompass  the  RPD  framework. 
The  results  of  our  simulations  suggest  that  decision  priming  (a  bias  toward  rendering  a 
given  decision  based  on  prior  information)  is  a  phenomenon  that  should  occur  in  many 
naturalistic  settings. 

Patterson,  R.  E.,  Pierce,  B.  J.,  Bell,  H.  H.,  Andrews,  D.,  &  Winterbottom,  M.  (2009).  Training 
robust  decision  making  in  immersive  environments.  Journal  of  Cognitive  Engineering 
and  Decision  Making,  3,  331-361. 
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We  provide  a  review  and  analysis  of  much  of  the  published  literature  on  decision  making 
that  is  relevant  to  the  design  of  immersive  environments.  This  review  draws  from  the 
basic  and  applied  literature  in  order  to  provide  insight  for  the  design  of  such  synthetic 
environments.  Included  in  this  review  are  articles  and  books  cited  in  other  works,  and 
articles  and  books  obtained  from  an  Internet  search.  Issues  discussed  are  (a)  an  overview 
of  immersive  decision  environments;  (b)  dual-process  decision  making;  (c)  training 
robust  intuitive  decision  making;  (d)  combining  analytical  and  intuitive  processing  in 
immersive  environments;  and  (e)  concluding  remarks.  For  the  development  of  robust 
decision  making  in  immersive  environments,  intuitive  reasoning  should  be  emphasized 
by  creating  an  immersive  situation  and  by  providing  for  the  development  of  automatic 
processing  through  implicit  learning,  with  the  latter  reinforced  by  explicit  thought 
processes.  Considerations  of  the  literature  on  decision  making  will  provide  insight  for 
future  design  solutions  for  immersive  decision  environments. 

Patterson,  R.  E.,  Pierce,  B.  J.,  Bell,  H.  H.,  &  Klein,  G.  (2009).  Implicit  learning,  tacit 

knowledge,  expertise  development,  and  naturalistic  decision  making.  Journal  of 
Cognitive  Engineering  and  Decision  Making,  4(4),  289-303. 

Implicit  learning  involves  the  largely  unconscious  learning  of  dynamic  statistical  patterns 
and  features.  This  kind  of  learning  is  a  ubiquitous,  robust  phenomenon  that  likely  occurs 
in  most,  if  not  all,  tasks  in  which  individuals  engage  throughout  their  lives.  In  this  paper, 
we  argue  that  implicit  learning  may  assist  in  the  acquisition,  retention,  and  transfer  of 
expertise  and  thus  provide  a  fonn  of  tacit  scaffolding  for  expertise  development.  The 
notion  of  implicit  scaffolding  represents  a  novel  and  interesting  area  of  future  research 
for  the  field  of  naturalistic  decision  making  and  naturalistic  cognition. 

Patterson,  R.,  Pierce,  B.,  Boydstun,  A.  S.,  Park,  L.  M.,  Shannan,  J.,  &  Tripp,  L.  (2010).  Implicit 
Learning  for  Developing  Intuitive  Decision  Making  Using  Immersive  and  Flat  Displays. 
Submitted  to  The  10th  International  Conference  on  Naturalistic  Decision  Making,  May 
31 -June  3,  2011,  Orlando,  FL. 

Intuition  can  be  a  critical  asset  when  making  decisions  in  various  situations.  In  this  study, 
we  investigated  the  acquisition  of  intuitive  decision  making  through  a  process  called 
implicit  learning.  During  training,  participants  passively  viewed  structured  sequences  of 
objects  or  letter  strings  presented  either  on  a  dynamic  terrain  seen  in  perspective  view  or 
on  a  static  flat  display.  We  also  had  participants  memorize  structured  strings  of  letters 
presented  on  a  static  flat  display.  Following  training,  participants  were  tested  for  implicit 
learning  by  making  intuitive  judgments  of  novel  structured  sequences  versus  random 
sequences.  Results:  implicit  learning  occurred  easily  across  all  conditions  and  there  was 
no  significant  performance  advantage  to  one  viewing  condition  over  another.  This  study 
has  implications  for  the  training  of  intuitive  decision  making  insofar  as  it  demonstrates 
that  cues  can  be  passively  and  effectively  acquired  from  exposure  to  different  artificial 
displays. 

Patterson,  R.E.,  Pierce,  B.J.,  Boydstun,  A.S.,  Park,  L.M.,  Shannan,  J.  &  Tripp,  L.M.  (2011). 

Implicit  Learning  for  Developing  Intuitive  Decision  Making  Using  Immersive  and  Flat 
Displays.  A  poster  presented  at  the  10th  International  Conference  on  Naturalistic  Decision 
Making,  Orlando,  FL,  June. 
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Intuition  can  be  a  critical  asset  when  making  decisions  in  various  situations.  In  this  study, 
we  investigated  the  acquisition  of  intuitive  decision  making  through  a  process  called 
implicit  learning.  During  training,  participants  passively  viewed  a  structured  sequence  of 
objects  or  letter  strings  presented  either  on  a  dynamic  terrain  seen  in  perspective  view  or 
on  a  static  flat  display.  We  also  had  participants  memorize  structured  strings  of  letters 
presented  on  a  static  flat  display.  Following  training,  participants  were  tested  for  implicit 
learning  by  making  intuitive  judgments  of  novel  structured  sequences  versus  random 
sequences.  Results:  implicit  learning  occurred  easily  across  all  conditions  and  there  was 
no  significant  performance  advantage  to  one  viewing  condition  over  another.  This  study 
has  implications  for  the  training  of  intuitive  decision  making  insofar  as  it  demonstrates 
that  cues  can  be  passively  and  effectively  acquired  from  exposure  to  different  artificial 
displays. 

Winterbottom,  M.,  Patterson,  R.,  Fournier,  L.,  Pierce,  B.  J.,  Williams,  L.,  &  Amann,  R.  (2009). 
Decision  Priming  in  an  Air-to-Ground  Attack  Decision  Scenario.  Poster  presentation  at 
the  NDM9,  the  9th  International  Conference  on  Naturalistic  Decision  Making  London, 
UK,  June  2009. 

Motivation  -  Validate  a  System  Dynamics  model  of  decision  speed  for  a  scenario 
relevant  to  an  Air  Force  mission  task.  Research  approach  -  Extend  a  decision  priming 
basic  research  paradigm  to  a  more  applied  case.  Findings/Design  -  Decision  priming  was 
found  to  occur  with  more  complex  stimuli  relevant  to  an  air-to-ground  target 
identification  scenario.  A  system  dynamics  model  was  developed  to  predict  those  results. 
Research  limitations/Implications  -  While  decision  delays  of  approximately  100 
milliseconds  were  found  here,  it  remains  to  be  determined  whether  decision  priming 
extends  to  decision  processes  extending  over  seconds  or  minutes.  Originality/Value  -  A 
temporally-based  method  of  modeling  a  decision  process  has  been  validated  for  an 
applied  air-to-ground  target  identification  scenario.  Many  other  decision  models  lack  this 
time  component  and  are  therefore  of  limited  use  in  time-critical  situations.  Take  away 
message  -  Tactical  situations  where  infonnation  is  processed  rapidly  and  sequentially  can 
result  in  decision  priming. 

Rehearsal  Enabling  Simulation  Technologies  (REST) 

Lerman,  D.  J.  (2010).  Correct  Weather  Modeling  of  non-Standard  Days  (10F-SIW-004).  In 
Proceedings  of  2010  Fall  Simulation  Interoperability  Workshop  (Fall  SIW)  SISO. 
Orlando,  FL:  SISO. 

Most  flight  simulators  compute  and  fly  in  a  weather  environment  that  matches  a  standard 
atmosphere,  with  the  possibility  of  adding  a  constant  temperature  or  pressure  offset  to  the 
standard  profile  to  train  non-standard-day  flight  conditions.  The  atmosphere  on  many 
simulators  is  computed  with  the  rate  of  change  of  pressure  altitude  against  geopotential 
altitude  being  1 .0,  regardless  of  any  temperature  increment  added  to  the  temperature 
profile.  Such  a  model  does  not  account  for  the  density  change  caused  by  temperature 
deviation  from  standard,  disobeys  the  laws  of  physics,  and  is  a  bad  model.  This  paper 
describes  the  effects  of  such  bad  modeling  on  training  tasks  and  presents  the  math  to 
model  the  atmosphere  correctly. 
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Sieverding,  M.  J.,  Stephens,  C.  W.,  &  Kent,  A.  E.  (2010).  The  Emerging  DoD  Requirement  for 
More  Realistic  Weather  in  Flight  Simulation.  In  Proceedings  of  the  2010  IMAGE  Society 
Conference;  July  2010;  Scottsdale,  AZ,  USA. 

This  paper  explored  the  history  of  the  requirements  and  technical  approaches  of 
implementing  weather  simulation  into  DoD  virtual  flight  simulators,  beginning  in  the  late 
1970s  and  extending  to  today’s  devices.  It  also  assesses  the  new,  emerging  requirements 
for  mission  rehearsal  and  networked  simulation  and  their  potential  for  deriving  the 
requirement  for  more  realistic  weather  simulation  fidelity.  Commercial  (Federal  Aviation 
Administration)  flight  simulator  weather  modeling  requirements  will  be  presented. 
Authoritative,  real-world  weather  data  sources,  models,  and  data  types  are  described.  A 
conceptual  design  for  implementing  them  in  modem  flight  simulators  is  presented 
including  visual  and  sensor  displays,  as  well  as  its  potential  impact  on  improved  training. 
This  paper  also  describes  the  rules  of  engagement  to  incrementally  prototype  the  addition 
of  common,  high  fidelity  weather  modeling  into  high  fidelity  flight  simulators.  The 
prototyping  goal  is  to  provide  Air  Force  DMO  with  a  blueprint  for  achieving  improved 
fidelity  and  expanded  training  capability  during  DMO  networked  operations.  The  focus 
of  this  paper  is  on  virtual  flight  simulation  applications,  but  ideas  and  methods  discussed 
here  could  be  applicable  to  virtual  ground-based  simulations  as  well,  especially  those  that 
may  be  networked  to  virtual  flight  simulations. 

Sieverding,  M.  J.,  Rybacki,  R.  M.,  &  Kent,  A.  E.  (2009).  A  Method  to  Compensate  for  Display 
System  Contrast  Ratio  Differences  in  Distributed  Simulation  (ADA525464).  In 
Proceedings  of  the  IMAGE  2009  Society,  St.  Eouis,  MO. 

A  considerable  challenge  facing  distributed  virtual  simulation  is  to  minimize  correlation 
differences  between  networked  simulations  so  that  humans-in-the-loop  perceive  and 
respond  to  the  same  stimulus--as  they  would  in  the  real  world.  There  are  many  causes  or 
domains  of  correlation  differences,  including  Appearance,  Behavior,  and  Time.  This 
paper  addresses  one  small  component  within  the  Appearance  domain  of  correlation. 

Considerable  work  has  been  done  across  the  Services  to  develop  methods  of  reusing 
environmental/spatial  datasets  that  not  only  reduce  the  schedule  and  cost  of  database 
generation,  but  also  achieve  greater  correlation  between  differing  simulations;  however, 
even  if  all  simulations  were  to  share  the  same  database  geometry,  textures,  colors,  and 
rendering  engine,  how  they  would  look  through  the  simulations’  different  display  systems 
can  vary  dramatically.  Distributed  simulation  may  include  many  different  virtual  entities 
that  are  outfitted  with  considerably  different  display  systems.  Each  display  system  type 
may  have  widely  varying  resolutions,  luminance,  and  contrast  ratios,  resulting  in  different 
perceived  scenes  even  if  all  systems  are  using  the  same  rendering  engine  and  spatial 
database. 

This  paper  presents  a  method  of  compensating  for  widely  varying  display  system  contrast 
ratios,  which  results  in  more  similarly  perceived  out-the-window  scenes  across  different 
simulations.  This  paper  presents  a  novel  algorithmic  approach  of  modifying  database 
colors  and  intensities.  The  principal  variable  within  the  algorithm  is  the  difference  in 
measured  display  system  contrast  ratios  between  two  simulator  systems.  Contrast  ratio 
test  methods,  tools,  and  results  are  also  presented  to  provide  objective  and  repeatable 
measures.  This  paper  also  describes  a  method  used  to  remap  all  pixel  colors  and 
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intensities  with  the  adjustment  algorithm  during  run-time,  using  plug-in  shader 
techniques. 

The  method  described  in  this  paper  offers  the  potential  for  application  across  any 
simulation  network  where  the  environment  model  is  built  from  common,  shared  datasets, 
where  different  types  of  display  systems  with  widely  varying  contrast  ratios  are 
employed,  and  where  correlated  (or  at  least  more  similar)  perceptions  are  required. 

Entity  Modeling  and  Full  Spectrum  Threat  Simulation 

Eidman,  C.,  Boyle,  G.  H.,  Lay,  M.,  &  Poliak,  E.  (2010).  Physics  Based  Modeling  of  a  Warhead 
Detonation  using  Graphics  Processors.  (Paper  No  10341)  In  Proceedings  of  2010 
Industry/Interservice  Training  Systems  Conference.  Orlando,  FL:  National  Security 
Industrial  Association. 

Modeling  highly  complex  entity  interactions  with  high  precision  is  challenging  in 
distributed  training  simulations.  When  using  existing  protocols  and  standards,  most 
weapons  effects  are  resolved  by  a  probability  of  a  kill  or  a  roll  of  the  dice,  reducing 
accuracy  and  fair  fight.  This  paper  presents  the  results  of  a  study  by  the  Air  Force 
Research  Lab  (AFRL)  to  determine  if  running  physics  algorithms  using  the  NVIDIA 
PhysX  SDK  with  a  constructive  simulation  could  more  accurately  model  the  damage 
associated  with  an  aircraft/missile  engagement  than  can  be  done  with  the  currently  used 
statistics  based  approach.  A  statistical  approach  assumes  damage  based  only  on  missile 
proximity  at  the  time  of  detonation  along  with  a  pre-determined  probability  of  kill.  This 
approach  typically  does  not  factor  the  exact  aspect,  geometry,  aircraft  material  strength, 
and  missile  warhead  capability  at  the  time  of  detonation.  The  study  leveraged  a  Physics- 
Based  Environment  Generator  (PBEG)  running  with  the  Expert  Common  Interactive 
Training  Environment  (XCITE)  constructive  simulation.  The  PBEG  was  programmed  to 
take  entity  control  of  the  aircraft  and  missiles  from  XCITE  to  determine  the  level  of 
damage,  provide  a  realistic  visualization  of  the  impact  and  return  damage  assessment  to 
XCITE.  A  target  aircraft  model  was  constructed  rendering  multiple  sections  and  facets, 
each  capable  of  individually  sustaining  damage.  A  missile  warhead  model  with  a 
simplified  TNT  explosive  and  fragmentation  was  integrated.  The  PBEG  computed  the 
force  of  the  initial  blast  shockwave  and  impact  from  each  high  velocity  shrapnel  piece 
striking  the  aircraft  on  each  specific  facet.  A  numerical  damage  value  is  assigned  for  each 
aircraft  part,  and  damage  results  are  returned  to  XCITE  for  training  feedback.  This 
approach  was  shown  to  increase  accuracy  in  damage  assessments  over  the  traditional 
approach.  This  paper  summarizes  the  study  results  and  provides  recommendations  for 
further  investigations  using  graphics  processors  for  modeling  weapons  effects. 

McCracken,  C.,  Breeden,  P.,  Eidman,  C.,  Walsh,  B.,  Williams,  L.,  Kamrowski,  M.,  Van  Ess,  G., 
&  Tygret,  K.  I.  (2010).  Integration  of  a  Proprietary  Missile  Server  for  Assessment  of 
Training  Research  Effectiveness.  In  Proceedings  of  2010  Inclustry/Interservice  Training 
Systems  Conference .  Orlando,  FL:  National  Security  Industrial  Association. 

The  Air  Force  Research  Laboratory,  Human  Effectiveness  Directorate,  Warfighter 
Readiness  Research  Division,  Mesa,  AZ  (711  HPW/RHA),  in  cooperation  with  Raytheon 
Missile  Systems,  Tucson,  AZ  (Raytheon),  developed  and  integrated  a  full-fidelity 
advanced  medium-range  air-to-air  missile  (AMRAAM)  weapon  server  into  a  high- 
fidelity  AFRL  F-16  flight  simulator.  The  project  goals  were  to  bring  manufacturer- 
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proprietary  missile  fly-out  performance  into  the  Air  Force  Distributed  Mission 
Operations  (DMO)  environment,  to  assess  simulation  effectiveness,  and  to  enhance 
operator  training.  Three  separate  AMRAAM  models  were  evaluated  and  tested  against 
operational  perfonnance  metrics  to  assess  capability  and  potential  training  impact:  the 
current  model  employed  by  the  Air  Force  F-16  Multi-Task  Trainer  cockpit,  the  model 
implemented  within  the  Air  Force  eXpert  Common  Immersive  Theater  Environment 
(XCITE)  synthetic  battlespace,  and  the  proprietary  AMRAAM  Raytheon  Simulation 
(ARS).  Multiple  parameters  were  compared,  such  as  maximum  range,  minimum  range, 
time  of  flight,  maneuvering  perfonnance,  and  target  intercept.  Testing  constraints  were 
overseen  and  validated  by  subject  matter  experts.  The  results  showed  that  successful 
integration  of  a  real-time  original  equipment  manufacturer-proprietary  missile  model 
with  Air  Force  DMO  assets  was  feasible.  Furthermore,  fly-out  testing  results  identified 
specific  parameters  and  situational  relationships  crucial  to  improving  warfighter 
instruction  during  brief  and  debrief.  This  performance  comparison  of  currently  employed 
weapons  models  with  the  Raytheon  AMRAAM  model,  highlighted  the  training 
effectiveness  available  through  the  careful  integration  of  manufacturer-proprietary 
modeling  and  simulation  tools. 

Audio  Integration/Demonstrations 

Knapp,  T.  D.  (2009).  Task  Order:  0037  Entity  Modeling  and  Immersive  Decision 

Environments:  CDRL  A022  Technical  Report  study /Services  Paragraph  4.4  Audio 
Integrations/Demonstrations.  Mesa  AZ:  Air  Force  Research  Laboratory,  Warfighter 
Readiness  Research  Division. 

This  report  detailed  the  Spatial  Audio  Integration  effort  for  Task  Order  0037  Entity 
Modeling  and  Immersive  Decision  Environments.  It  describes  the  approaches 
considered,  problems  encountered,  their  resolutions,  and  provides  implementation  details. 
There  is  also  a  summary  of  current  limitations  and  suggestions  for  future  improvements. 

Ultra-High  Resolution  Deployable  Projector  Components  for  Simulation 

Geri,  G.  A.,  &  Williams,  L.  A.  (2011).  Photometric  and  perceptual  evaluation  of  the  spatial 
resolution  and  laser-speckle  properties  of  a  Micro  Vision  ShowWX  Pico  projector 
(Warfighter-Contract  Technical  Memorandum,  TO37(201 1)-01).  Mesa  AZ:  Air  Force 
Research  Laboratory,  Human  Effectiveness  Directorate,  Warfighter  Readiness  Research 
Division. 

The  spatial  and  temporal  characteristics,  as  well  as  the  laser-speckle  properties,  of  a 
Micro  Vision  ShowWX  Pico  Laser  Projector  were  evaluated  using  standard  techniques. 
One  purpose  of  the  evaluation  was  to  detennine  if  a  relatively  inexpensive,  commercially 
available  laser  projector  is  suitable  for  use  as  a  flight- simulator  cockpit  display.  More 
generally,  the  evaluation  was  designed  to  implement  and  assess  techniques  for  estimating 
spatial-resolution  and  speckle-contrast,  and  to  determine  whether  those  techniques  may 
be  useful  in  either  future  laser-projector  evaluations,  or  as  part  of  a  laser-display 
measurement  standard. 

Geri,  G.  A.,  &  Williams,  L.  A.  (2011).  Perceptual  assessment  of  laser-speckle  contrast. 
Manuscript  submitted  to  a  peer-reviewed  displays  journal. 
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A  procedure  is  described  for  perceptually  assessing  the  effects  of  laser-speckle  on  target 
discrimination  in  the  context  of  developing  a  laser-display  measurement  standard.  The 
perceptual  approach  avoids  the  difficulties  inherent  in  assessing  speckle  contrast 
photometrically.  Size  thresholds  for  triangular  test  stimuli  were  obtained  using  an 
orientation  discrimination  task.  Data  were  obtained  for  both  laser  and  LCD  displays, 
using  both  a  method  of  constant  stimuli  and  a  QUEST  procedure.  The  identification 
threshold  was  29%  higher  for  the  laser  display.  Given  that  the  measured  spatial 
resolutions  of  the  displays  were  similar,  the  difference  in  threshold  was  attributed  to  the 
effects  of  laser  speckle.  Laser-speckle  size  was  also  estimated  as  part  of  the  display 
calibration.  For  a  chosen  camera  f-number,  speckle-size  data  are  necessary  to  determine 
the  sensor  pixel-density  required  to  photometrically  estimate  speckle  contrast,  and  they 
may  be  useful  for  comparison  with  the  data  obtained  using  the  perceptual  procedure 
described  here.  Mean  speckle  size  was  estimated  by  scaling  and  normalizing  the  laser- 
speckle  luminance  and  then  detennining  the  width  of  the  associated  autocorrelation 
function.  It  is  proposed  that  a  laser-display  standard  be  developed  based  in  part  upon  a 
comparison  of  task  performance  on  the  laser  display  to  be  evaluated  and  on  a  standard 
display. 

Lloyd,  C.  J.,  Williams  L.  A.,  &  Pierce,  B.  J.  (2011).  A  model  of  the  relative  effects  of  key  task 
and  display  design  parameters  on  training  task  perfonnance.  In  Proceedings  of  the  2011 
IMAGE  Society  Conference;  June  2011;  Scottsdale,  AZ,  USA. 

This  paper  describes  the  design  and  initial  validation  of  a  model  of  the  relative  effects  of 
key  task  and  display  design  parameters  on  training  task  performance.  The  display  system 
design  parameters  include  pixel  hold  time,  pixel  pitch,  luminance,  contrast,  and  noise. 

The  task  and  observer  parameters  include  target  size  and  range,  angular  velocity  (of  the 
image),  target  contrast,  and  observer  capability  (e.g.,  acuity). 

This  model  was  developed  as  part  of  the  Immersive  Display  Evaluation  and  Assessment 
Study  (IDEAS)  program  for  the  USAF.  The  model  was  designed  to  be  used  by  display 
system  acquisition  professionals  who  must  develop  defensible  display  system 
requirements  that  are  based  on  the  ability  of  the  display  system  to  support  the  training 
planned  for  the  system.  The  model  was  also  designed  for  those  display  systems 
professionals  in  the  supplier  community  who  wish  to  steer  their  product  designs  in  the 
direction  of  maximum  utility  to  the  USAF. 

The  initial  validation  results  indicate  the  model  accurately  summarizes  the  findings  from 
several  published  evaluations  that  employed  tasks  generally  representative  of  flight 
simulation  training.  The  model  was  then  used  to  make  specific  predictions  of  the  effects 
of  220  combinations  of  four  parameters  (hold  time,  velocity,  pitch,  and  luminance)  on  the 
range  at  which  pilots  could  reliably  identify  fighter  sized  aircraft.  A  fonnal  evaluation  of 
the  effects  of  these  parameters  was  conducted  and  the  results  of  this  evaluation  are 
compared  with  the  predictions.  A  high  correlation  between  model  predictions  and  the 
results  of  this  evaluation  was  obtained. 

Lloyd,  C.  J.,  Joralmon,  D.  Q.,  Amann,  R.  T.,  Tai,  C.  F.,  Williams  L.  A.,  &  Pierce,  B.  J.  (2011). 
Relative  effects  of  five  display  design  variables  on  aircraft  identification  range  in 
daylight.  In  Proceedings  of  the  2011  IMAGE  Society  Conference;  June  2011;  Scottsdale, 
AZ,  USA. 
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This  paper  summarizes  the  findings  from  two  human  factors  evaluations  conducted  as 
part  of  the  Immersive  Display  Evaluation  and  Assessment  Study  (IDEAS)  program.  For 
both  evaluations  experienced  USAF  F-16  pilots  discriminated  and  positively  identified 
distant  fighter-sized  aircraft.  On  each  trial  the  ownship  rapidly  approached  a  pair  of 
aircraft,  one  -Triend”  and  one  -Toe,”  and  the  observers  designated  the  foe  as  quickly  and 
accurately  as  they  could. 

The  first  evaluation  focused  on  variables  expected  to  be  primary  determinants  of  motion- 
induced  blurring  (e.g.,  hold  time  and  angular  velocity)  for  modern  display  systems.  The 
second  evaluation  filled  out  the  data  set  required  to  validate  a  more  complete  model  of 
the  design  variables  expected  to  mediate  task  perfonnance  for  very  high  resolution 
display  systems.  Across  the  two  evaluations,  task  performance  was  measured  as  a 
function  of  420  combinations  of  five  practical  display  system  design  variables  including: 
pixel  hold  time,  angular  velocity  of  the  image,  pixel  pitch  (resolution),  display  contrast, 
and  display  luminance. 

Prior  to  conducting  the  evaluations  a  computational  model  was  prepared  and  used  to 
make  quantitative  predictions  of  the  effects  of  these  design  variables.  The  correlation 
between  the  model  predictions  and  the  results  of  the  first  evaluation  was  high  (e.g.,  R2  > 
0.75, p  <  0.001,  109  df).  After  tuning  three  parameters  in  the  model  to  the  data  the 
correlation  increased  significantly  (R2  =  0.973,  p  <  0.001,  106  df). 

A  significant  benefit  provided  by  the  model  is  the  quantification  of  the  interactions 
among  the  design  variables.  Thus,  the  model  is  useful  for  examining  the  impact  of  design 
trades  among  the  variables  that  affect  task  performance. 

Lloyd,  C.  J.  (2011).  Towards  a  decision  support  system  for  generating  display  requirements  for 
simulation  training.  In  Proceedings  of  2011  Industrv/Interservice  Training  Systems 
Conference.  Orlando,  FL:  National  Security  Industrial  Association. 

The  complexity  of  the  acquisition  process  for  training  display  systems  has  increased  in 
recent  years  due  to  the  rapid  changes  in  the  technologies  underlying  these  systems.  For 
two  decades  prior  to  this  complexity  surge,  the  relationship  between  the  military 
customer  and  their  suppliers  was  more  stable:  most  suppliers  provided  custom  image 
generators  and  >90%  of  display  systems  used  mature  CRT  projectors.  Display  system 
customers  fell  into  the  habit  of  essentially  adopting  current  product  specifications  as  the 
requirements”  for  new  systems,  a  practical  strategy  given  these  systems  had  worked 
well  for  many  years.  The  recent  introductions  of  multiple  new  technologies  (e.g.,  PC  Igs, 
COTS  projectors,  automated  display  calibration  systems,  and  practical  stereoscopic 
displays)  have  substantially  disrupted  the  traditional  relationships  between  customer  and 
supplier.  New  suppliers  have  entered  the  system  integration  market  as  the  incumbents 
have  struggled  to  keep  up  with  technology  changes.  Customers  can  no  longer  rely  on 
relatively  stable  product  capabilities  or  long  term  supplier  relationships  to  ensure  their 
next  training  display  system  meets  the  needs  of  their  users. 

In  response  to  these  trends,  the  Immersive  Display  Evaluation  and  Assessment  Study 
(IDEAS)  program  was  initiated  in  the  summer  of  2010.  The  long  term  goal  of  this 
program  is  to  produce  a  decision  support  system  (DSS)  that  will  enable  (1)  Air  Force 
acquisition  professionals  to  rapidly  generate  defensible  display  system  requirements 
given  a  set  of  training  task  requirements,  (2)  suppliers  to  develop  new  visual  systems  and 
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prepare  proposals,  and  (3)  aid  certifiers  to  measure  perfonnance.  Some  overarching 
goals  for  the  DSS  include: 

Usable  by  persons  who  understand  the  training  need  but  are  not  experts  in  current  display 
technology 

•  Produces  defensible  requirements  based  on  task  performance  data  and  previously 
validated  models 

•  Identify  the  data  and  models  that  underlie  each  recommendation 

•  Allows  high  level  design  trades  to  be  made  during  the  requirements  generation 
process 

•  Produces  the  specific  text  and  levels  for  selected  display  system  requirements 

•  Provides  the  approved  measurement  procedure  for  each  requirement 

•  Employs  an  architecture  that  allows  updating  the  data  and  models  as  technologies 
evolve 

Development  and  validation  of  the  first  model/data  to  be  incorporated  into  the  DSS  began 
in  the  fall  of  2010.  This  work  is  summarized  in  DTIC  technical  reports  and  two  papers 
presented  at  IMAGE  2011.  The  present  paper  provides  a  brief  overview  of  the 
development  and  validation  of  the  model  and  focuses  primarily  on  how  the  model  and 
DSS  might  be  used  by  stakeholders. 

The  paper  and  presentation  will  describe  the  general  capabilities  of  the  DSS  and  will 
provide  concrete  examples  of: 

•  Generation  of  selected  display  requirements  from  the  F-16  MTC  training  task  list 

•  Use  of  the  DSS  to  make  source  selection  decisions 

•  Generation  of  measurement  procedures 

•  Use  of  the  DSS  by  a  supplier  for  product  development  planning 

A  primary  goal  of  this  paper  is  to  solicit  inputs  from  stakeholders  prior  to  and 
concurrently  with  the  development  of  the  system. 

Lloyd,  C.  J.,  Joralmon,  D.  Q.,  Amann,  R.  T.,  Tai,  C.  F.,  Morgan  W.,  Geri,  G.  A.,  Williams  L.  A., 
&  Pierce,  B.  J.  (2011).  Effects  of  hold  time,  angular  velocity,  pitch,  and  luminance  on 
simulated  aircraft  identification  range.  AFRL  technical  report,  to  be  submitted  to 
Defense  Technical  Information  Service  (DTIC). 

This  paper  summarizes  the  findings  from  the  first  of  two  human  factors  evaluations 
conducted  as  part  of  the  Immersive  Display  Evaluation  and  Assessment  Study  (IDEAS) 
program.  In  this  evaluation  experienced  USAF  F-16  pilots  discriminated  and  positively 
identified  distant  fighter-sized  aircraft.  On  each  trial  the  ownship  rapidly  approached  a 
pair  of  aircraft,  one  -friend”  and  one  -Toe,”  and  the  observers  designated  the  foe  as 
quickly  and  accurately  as  they  could. 

This  evaluation  focused  on  the  effects  of  three  variables  expected  to  be  primary 
detenninants  of  motion-induced  blurring;  hold  time,  angular  velocity  of  the  image,  and 
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pixel  pitch.  An  external  motion  blur  reduction  shutter  was  used  to  systematically 
manipulate  hold  time  causing  luminance  to  co-vary  with  hold  time.  Thus,  a  fourth 
independent  variable,  luminance,  was  included  in  the  evaluation  so  that  its  effect  could  be 
estimated  independently  of  the  hold  time  variable. 

Prior  to  conducting  the  evaluation,  a  computational  model  was  prepared  and  used  to 
make  quantitative  predictions  of  the  effects  of  these  design  variables.  The  correlation 
between  the  model  predictions  and  the  results  of  this  evaluation  was  high  (e.g.,  R2  > 

0.75, p  <  0.001,  109  df).  After  tuning  three  parameters  in  the  model  to  the  data,  the 
correlation  increased  significantly  (R2  =  0.973,  p  <  0.001,  106  df). 

A  significant  benefit  provided  by  the  model  is  the  quantification  of  the  interactions 
among  the  design  variables.  Thus,  the  model  is  useful  for  examining  the  impact  of 
design  trades  among  the  variables  that  affect  task  performance. 

A  summary  of  this  evaluation  was  published  at  the  IMAGE  2011  conference.  The 
present  report  contains  more  of  the  details  of  the  evaluation,  the  instructions  to  observers, 
and  a  table  of  the  mean  response  data  for  the  220  experimental  conditions. 

Lloyd,  C.  J.,  Joralmon,  D.  Q.,  Amann,  R.  T.,  Tai,  C.  F.,  Morgan  W.,  Williams  L.  A.,  &  Pierce,  B. 
J.  (2011).  Effects  of  luminance,  contrast,  hold  time,  angular  velocity,  and  pitch,  on 
simulated  aircraft  identification  range.  AFRL  technical  report,  to  be  submitted  to 
Defense  Technical  Information  Service  (DTIC). 

This  paper  summarizes  the  findings  from  the  second  of  two  human  factors  evaluations 
conducted  as  part  of  the  Immersive  Display  Evaluation  and  Assessment  Study  (IDEAS) 
program.  In  this  evaluation  experienced  USAF  F-16  pilots  discriminated  and  positively 
identified  distant  fighter-sized  aircraft.  On  each  trial  the  ownship  rapidly  approached  a 
pair  of  aircraft,  one  -Triend”  and  one  -foe,”  and  the  observers  designated  the  foe  as 
quickly  and  accurately  as  they  could. 

The  first  evaluation  focused  on  the  variables  expected  to  be  primary  determinants  of 
motion-induced  blurring  (e.g.,  hold  time  and  angular  velocity)  for  sample-and-hold 
display  systems.  This  second  evaluation  filled  out  the  data  set  required  to  validate  a  more 
complete  model  of  the  design  variables  expected  to  mediate  task  perfonnance  for  very 
high  resolution  display  systems.  In  this  evaluation,  task  performance  was  measured  as  a 
function  of  200  combinations  of  five  practical  display  system  design  variables  including: 
display  luminance,  display  contrast,  pixel  hold  time,  angular  velocity  of  the  image,  and 
pixel  pitch  (resolution). 

Prior  to  conducting  the  evaluation,  a  computational  model  was  prepared  and  used  to 
make  quantitative  predictions  of  the  effects  of  these  design  variables.  The  correlation 
between  the  model  predictions  and  the  results  of  this  evaluation  was  high  (e.g.,  R2  = 

0.91 1,  p  <  0.001,  199  df).  The  model  parameters  have  not  yet  been  tuned  to  the  data 
collected  in  this  evaluation. 

A  significant  benefit  provided  by  the  model  is  the  quantification  of  the  interactions 
among  the  design  variables.  Thus,  the  model  is  useful  for  examining  the  impact  of 
design  trades  among  the  variables  that  affect  task  performance. 


53 

DISTRIBUTION  A.  Approved  for  public  release;  distribution  unlimited.  (Approval  given  by 
88  ABW/PA,  88ABW-201 2-1 090,  5  Mar  1 2.) 


A  summary  of  this  evaluation  was  published  at  the  IMAGE  2011  conference.  The  present  report 
contains  more  of  the  details  of  the  evaluation  and  a  table  of  the  mean  response  data  for  the  200 
experimental  conditions. 
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LIST  OF  SYMBOLS,  ABBREVIATIONS,  AND  ACRONYMS 


2D 

3D 

6D0F 

ACE 

AFCD 

AFRL 

ASC 

ATC 

CAF 

CGF 

CM 

CRADA 

DE 

DIS 

DISNET 

DMO 

DSS 

DTED 

EDT 

EMTS 

ESLP 

ESRI 

FOB 

GARS 

GeoTIFF 

GUI 

HLA 

HUD 

IDEAS 

IFF 

IG 

l/ITSEC 

IOS 

IR 


Two-Dimensional 

Three  Dimensional 

6-degrees  of  freedom 

Advanced  Command  Environments 

Air  Force  Common  Dataset 

Air  Force  Research  Laboratory 

Aeronautical  Systems  Center 

Air  Traffic  Control 

Combat  Air  Force 

Computer  Generated  Force 

Configuration  Management 

Cooperative  Research  and  Development  Agreement 
Directed  Energy 

Distributed  Interactive  Simulation 

Distributed  Interactive  Simulation  Network 

Distributed  Mission  Operations 

Decision  Support  System 

Digital  Terrain  Elevation  Data 

Employment,  Doctrine,  and  Tactics 

Entity  Modeling  and  Full  Spectrum  Threat  Simulation 

Evans  and  Sutherland  Laser  Projector 

Environmental  Systems  Research  Institute 

Forward  Operating  Base 

Global  Area  Reference  System 

Geospatial  Tagged  Imagery  File  Format 

Graphical  User  Interface 

High  Level  Architecture 

Head  Up  Display 

Immersive  Display  Evaluation  and  Assessment  Study 
Identification  Friend  or  Foe 
Image  Generator 

Interservice/Industry  Training,  Simulation  and  Education  Conference 

Instructor  Operator  Station 

Infrared 
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IRB 

JTAC 

LVC 

M&S 

MEMS 

MGRS 

MIT 

MTC 

NAVAIDS 

NGTS 

NIU 

PBEG 

PDU 

REST 

RFTE 

RIDER 

RPD 

SE 

SME 

TACAN 

TRS 

UN 

USAF 

UTM 

WRDS 

WRSTP 

XCITE 


Institutional  Research  Board 
Joint  Terminal  Attack  Controller 
Live-Virtual-Constructive 
Modeling  and  Simulation 
Micro-Electrical-Mechanical  Systems 
Military  Grid  Reference  System 
Massachusetts  Institute  of  Technology 
Mission  Training  Center 
Navigational  Aid  Systems 
Next  Generation  Threat  System 
Network  Interface  Unit 
Physics-Based  Environment  Generator 
Protocol  Data  Unit 

Rehearsal  Enabling  Simulation  Technologies 

Radio  Frequency  Threat  Environment 

Robust  Immersive  Decision  Environments  Research 

Recognition-Primed  Decision 

Synthetic  Environment 

Subject  Matter  Expert 

Tactical  Air  Navigation  Systems 

Training  and  Rehearsal  System 

United  Nations 

United  States  Air  Force 

Universal  Transverse  Mercator 

Western  Ranges  Dataset 

Warfighter  Readiness  Science  &  Technology  Program 
eXpert  Common  Immersive  Theater  Environments 


56 

DISTRIBUTION  A.  Approved  for  public  release;  distribution  unlimited.  (Approval  given  by 
88  ABW/PA,  88ABW-201 2-1 090,  5  Mar  1 2.) 


